Psychosis is one of the most disabling mental health conditions presenting with significant distress, suicidal ideation, impaired social and occupational functioning, and physical ill-health [1,2]. Paranoia and associated delusions are common symptoms of psychosis, are associated with more distress than other types of delusion [3], are most likely to be acted upon [4], and represent a strong predictor of hospitalization [5]. In the United Kingdom, over one-third of patients with psychiatric conditions experience paranoia, which also presents in a range of other psychopathologies such as depression [6], bipolar disorder [7], posttraumatic stress disorder [8], anxiety [9], as well as schizophrenia [10].

The National Institute for Health and Care Excellence recommended cognitive behavioral therapy (CBT) for treating psychosis. CBT, however, is received by only 1 in 10 of those who could benefit and has shown only moderate effect sizes for the treatment of delusions [11,12], although effect sizes are higher for those studies targeting delusions specifically, as opposed to generic CBT [13,14]. Unfortunately, a significant proportion of patients having paranoia continue to experience distressing symptoms following psychological treatment [15,16]. Consequently, there is a need for novel, highly accessible, and low-cost interventions for paranoia, either as standalone treatments or as adjuncts to boost existing therapies. Cognitive bias modification (CBM) is a class of intervention that may address these needs.

The class of CBM interventions works on the premise that cognitive bias is a putative causal factor of various mental health concerns [17-21]. One form of cognitive bias is interpretation bias, which is the tendency for individuals to think about a situation in a negatively skewed direction. However, the same situation could also be interpreted in a benign or positive direction. Repeated negatively biased interpretations are thought to contribute to the formation and maintenance of psychological symptoms and increase distress [3]. Across many studies, researchers have found evidence of interpretation bias among anxiety [19], depression [20], and social phobia [21], with some work on interpretation bias in paranoia [3,22-25].

CBM is a class of targeted treatment that focuses on manipulating naturally occurring interpretation bias in a more helpful direction, with findings from many studies demonstrating the positive efficacy of CBM with various psychiatric disorders, including anxiety, affective disorders, and substance addictions [26-28]. There are several benefits to CBM. First, CBM can be self-administered and disseminated over numerous settings [29], thereby reducing the need for mental health professionals. Next, CBM has the potential to benefit patients whose symptoms may influence their trust in a therapist [30]. Third, CBM can be delivered on a digital platform, which means that it is highly accessible at a low cost [31,32].

Despite these benefits and the positive efficacy of CBM with various mental health concerns, there is a dearth of studies on CBM that address psychosis, with only some preliminary evidence of the feasibility and implications of this approach. For example, Steel et al [33] demonstrated the effects of CBM on anxiety in individuals diagnosed with schizophrenia. The results from that study showed that a subgroup of participants exhibited positive changes in interpretation bias. Turner et al’s [34] case study on patients who experienced social anxiety following a psychotic episode demonstrated similar positive changes in interpretation bias. In a feasibility study, Yiend et al [35] directly examined the effects of CBM in patients with paranoia, using an intervention called CBM for paranoia (CBM-pa). In that study, 63 participants with clinically significant persecutory or paranoid symptoms were randomly assigned to either the CBM-pa group (n=32) or the control group (n=31). Participants in the CBM-pa group were presented with 40 short passages over 6 weekly sessions using a software called E-prime (Psychology Software Tools, Inc). Users were invited to complete the final word of each passage, which contained missing letters. Once completed, the word resolved the ambiguity of the passage in a benign nonparanoid manner. A follow-up yes or no question reinforced the benign interpretation of the passage (see Figure 1). The sessions were self-directed as users completed each word task independently on the computer. The control group received the same number of sessions over 6 weeks that included items of general knowledge and facts and everyday activities. Results showed that relative to the control group, participants in the CBM-pa group showed larger reductions in negative interpretation bias and paranoid symptoms.

Each passage of CBM-pa depicted an emotionally ambiguous scenario, all of which were developed with a user-centered approach, by inviting living experts and experienced clinicians to review all training materials to ensure the clinical relevance of the items to paranoia.

Researchers have shown that people experiencing psychosis can benefit from digital therapeutics, but despite the wide availability of digital therapeutics on the app market, many have insufficient evidence-based data to support their efficacy, design, and development [36]. It is important to take a user-centered development approach to design user-friendly, engaging, and self-managing digital therapeutics for psychosis [37,38] by involving multiple collaborators, including service users, researchers, and the design team. This approach is known to increase the adoption of the app by end users [38] and improve app design and content [39,40]. Self-administered mobile health apps without quality evidence-based data to support their use may decrease the usability and effectiveness of the treatment [41]. This is important for both app design as well as the intervention content. Researchers have demonstrated that biases are stronger when the encountered situation aligns with the individual's common everyday experiences [42,43]. Yiend et al [35] used content-specific training materials for paranoia to capture and modify paranoia interpretation bias commonly experienced by patients with paranoid symptoms. Content materials were co-designed with relevant contributors, and sessions were presented in rank order of increasing severity of items using Freeman et al’s [44] hierarchy of paranoia as a guide. The training items covered 6 categories relevant to paranoia: social or interpersonal threat, delusions of reference or magical thinking, the threat of persecution or spying, general suspiciousness or distrust, medical or paramedical or health care threat, and physical harm.

Building on from Yiend et al [35] and following a user-centered development approach, we aimed to develop CBM-pa into a 12-session mobile app therapeutic called STOP (Successful Treatment of Paranoia). As a part of an ongoing clinical trial, we are testing STOP’s efficacy, we tested STOP’s efficacy against the control group. STOP included the original item content from the CBM-pa feasibility study and newly developed items for 6 additional training sessions (details of content development for the 6 training sessions from the CBM-pa feasibility study will be reported separately). In this paper, we reported the detailed development process of STOP, which had the following objectives: (1) take a user-centered approach with input from living or lived experts, clinicians, and academics to create and evaluate paranoia-relevant item content to be used in STOP and (2) engage with living or lived experts and the design team from Avegen to cocreate and pilot-test the STOP mobile app prototype. Avegen is a digital health care company specializing in developing innovative health care technologies [45].

The methodology of the STOP development process involved (1) 4 stages for objective 1: text creation, text evaluation, graphics development, and graphics evaluation and (2) 1 stage for objective 2—STOP mobile phone app usability testing.

Objective 1 was intended to ensure clinical relevance, content specificity to paranoia, face validity of the training materials, and user acceptability for STOP. Objective 2 provided data on living or lived experts’ perspectives on the functionality, interface, and acceptability of the prototype STOP app to reveal areas of strength and those that needed improvement.

The process of material development and testing spanned 12 months and involved extensive iterative input from (1) living or lived experts, (2) clinical psychologists, and (3) the STOP academic research team (see Multimedia Appendix 1 [46,47] for inclusion criteria of each contributor). Input from each contributing group and the numbers varied according to the task required. Figure 2 shows a schematic of the development process.

The STOP trial program of work received ethical approval from the London-Stanmore Research Ethics Committee (reference 21/LO/0896), and all those participating in the work described gave consent for publication.

This study focused on the development of new material to be used in STOP—a novel mobile phone app designed to reduce the symptoms of paranoia. This self-administered digital therapeutic aims to reduce symptoms by presenting everyday ambiguous situations that can trigger paranoid thoughts and then normalizing users’ interpretations of these situations. However strong the conceptual basis of a new therapeutic, its quality, acceptability, and efficacy will be dependent upon its detailed content, input, and recommendations from various relevant contributors [38,40]. This is especially true for interventions that are based on CBM methods, which rely solely on content for their effect [24,25], and interventions that address psychosis [37,39]. The work presented in this paper represents a 12-month activity with clinicians, living or lived experts, a digital solutions design team, and researchers to develop and evaluate the therapeutic content of the mobile app STOP. Specifically, the co-design approach represents a thorough attempt to achieve our two objectives, which is to (1) take a user-centered approach to create and evaluate paranoia-relevant CBM item content and (2) engage with living or lived experts and the digital solutions design team to create and pilot test the STOP mobile app prototype.

For all training materials, we reached a priori-defined acceptable threshold for all rating criteria: paranoia severity and readability of the scenarios, and there were no systematic differences in item length between intervention and control content nor within the 6 newly created sessions of STOP. These data were used to inform the progression of the therapeutic intervention by arranging session content in order to increase paranoia severity. To reflect clinician-administered cognitive therapies, a “drill-down” approach from surface-level automatic thoughts to more profound core beliefs was adopted across sessions by using selected specific verbs to reflect on each level of thought process. For item graphics, we also found no systematic differences in users’ ratings of complexity, attractiveness, and interest between intervention and control groups. Furthermore, evaluations from 2 pilot tests of STOP with living or lived experts showed that user ratings were above our a priori acceptable thresholds for all evaluated features of the mobile app, suggesting that users found the STOP app easy to use and navigate, suitably interactive, helpful, and secure.

The existing literature demonstrates the importance of co-designing mobile phone apps for mental illnesses with multiple collaborators [37-39]. This work illustrates 1 approach to implementing a detailed user-centered development process that was applied throughout the entire design and development process of a mobile app. This may serve as a useful model for others, as the field of digital mental health continues to grow exponentially. Our co-design is likely to have improved the relevance, authenticity, face validity, and acceptability of both the therapy interface and its content compared to a researcher-led approach, although we cannot provide direct evidence on this. In each phase of STOP’s creation, we involved relevant contributors to provide feedback, open discussion, and formal usability testing of STOP’s content and mobile app. Contrary to STOP’s predecessor CBM-pa [35] where only the researchers designed training materials, in this work, we refined both the therapeutic content (training material) and the mobile app implementation, following contributors’ recommendations. The literature on co-design suggests that the careful and inclusive development process we have followed is likely to enhance user engagement and uptake of STOP [38]. There is also evidence that co-design improves treatment adherence and motivation [51]. Further additional features that we have included, such as graphical enhancement, use of therapeutic content based on actual patient experience, and close attention to the reduction in potential confounding variables (eg, time spent in therapy could inadvertently influence the “dose” of a session), may improve the intervention when tested against a control group in a clinical trial.

There are several improvements that could be made to this study. First, despite basing content development on user-generated examples, the generalization of these examples is limited to the individuals that generated them. Future work should consider ways to tailor content to the individual in real time or prior to the start of therapy. The development of personalized predictive algorithms and agile methods of therapeutic content selection will be one way to do this. Second, it will be important to test the STOP app for acceptability and feasibility of usage in a live clinical service setting, as mentioned by national organizations such as The National Institute for Health and Care Excellence [52]. By the same token, our small sample of raters was recruited from single clinical service units within the United Kingdom, thereby limiting the representativeness of the feedback and ratings received.

Third, there are several limitations relevant specifically to a clinical trial context use of STOP, as opposed to real-world deployment. For example, we only matched items by length between experimental and control groups, as measured by items’ character counts; a more thorough matching process would likely reduce any further confounding effects of the training material. Using single factors such as these to control for arbitrary effects of the intervention is limited, and in the future, other factors could be added to better control for confounds (eg, measuring actual reading speed, user’s comprehension of items, gender-specific content, and intercultural relevance).

A further trial-related limitation is that graphics were rated on only 3 rating criteria pertaining to visual appearance, which were derived from subscales of a standardized instrument. The limited selection of scales was a pragmatic decision, and future work could match graphic content on a wider range of criteria, for example, including aspects of appearance, such as aesthetics, excitement, likeability, and so on, all of which are included in the original instrument that was used to motivate our selection of scales. In addition to graphic enrichments, other elements, including badges, progress trackers, and trivia, are integral to the STOP mobile app and are derived from earlier focus group discussions, but these have not been evaluated. Ideally, all enrichments should be tested systematically to determine their effectiveness in engaging and motivating service users.

Finally, although we rely on feasibility data and previous ratings and feedback [25] to validate the first 6 sessions of STOP, nevertheless, an improvement in future work would be to evaluate all 12 sessions simultaneously on the same metrics.

In conclusion, CBM-pa is a relatively recent novel psychological intervention that has now been extended into the digital therapeutic called STOP. Material development and app development for any new CBM content should follow an iterative and rigorous process involving multiple contributors that include living or lived experts, researchers, clinicians, and the design team. This user-centered approach to intervention development maximizes the relevance of therapeutic content to the target user group. In so doing, researchers will most likely also optimize user acceptability, effectiveness, and engagement to create the best possible mobile health interventions for people with severe psychiatric disorders.