This was a formative, qualitative usability study in which participants from the target population of interest completed usability testing of an existing mental health app. Usability testing occurred in 2024, and data analysis occurred in 2024 and 2025. Of note, additional feedback from study participants regarding contextual determinants of engagement with mental health apps was reported separately [12].

This study was performed via an academic-industry partnership. Academic investigators had full independence in designing the study, recruiting participants, conducting the usability testing, performing data analysis, and reporting study results. The industry partner provided access to the mental health app under investigation and created mock-ups of refined app screens and workflows for the iterative usability testing process. The academic investigators shared the study findings with the industry partner so that the industry partner could refine its mental health app accordingly.

Ethical approval was obtained by the Washington University Institutional Review Board via a waiver of written informed consent prior to participant recruitment (Institutional Review Board ID 202311024). Participant data were maintained in secure databases and were deidentified before sharing outside the institutional review board–approved study team. Participants received a US $30 gift card for participation.

Participants were at least 45 years old, reported at least moderate depression or anxiety symptoms (Patient Health Questionnaire-9 ≥ 10, Generalized Anxiety Disorder-7 score ≥ 10, or both), and endorsed chronic pain (defined as pain on most days or every day in the past three months) [20,39-41].

Exclusion criteria included endorsement of an active mental health crisis that warranted escalation of care (eg, active suicidal ideation and psychosis), cognitive impairment that would interfere with communication with the researcher or meaningful engagement with the mental health app (assessed informally by the research coordinator asking the potential participant to restate the study purpose, activities, and risks, benefits, and alternatives of study participation during the informed consent process), and residing outside the United States.

Participants were a self-selected convenience sample recruited from multiple sources including a chronic musculoskeletal pain clinic at a US academic medical center, email, and social media advertisement via the medical center’s volunteer research participant registry, and targeted advertisement in a free online health community (The Mighty). Eligibility screening was performed using a self-reported questionnaire hosted by REDCap (Research Electronic Database Capture; Vanderbilt University), followed by verification by the research coordinator [42,43]. Recruitment was closed when usability testing by additional participants no longer yielded new themes within the convenience sample during the iterative analysis process [44].

The mental health app assessed in usability testing, Wysa for Chronic Pain, aims to address mental health and chronic pain through a multifaceted approach by facilitating behavioral activation, pain acceptance, and improved sleep quality [45-48]. It guides users through evidence-based cognitive behavioral therapy, mindfulness-based interventions, and sleep support. The app has used a rule-based conversational agent with natural language understanding to deliver therapeutic content since 2016. In 2024, it augmented this approach with the addition of large language models at specific nodal points. Large language model–based capabilities were integrated into check-in conversations to improve the chatbot’s ability to understand context-specific content over multiple interactions and respond in a more personalized manner to the user. However, risk-based and crisis conversations have remained completely rule-based and written by a clinician. Although the parent Wysa company offers multiple commercially available products, Wysa for Chronic Pain is not currently commercially available.

The therapeutic tools within Wysa for Chronic Pain are designed to be delivered via a combination of a structured curriculum and optional user-led exploration. By default, the app sends push notifications to users twice daily. The morning notification prompts an activity scheduling and planning task, and the evening notification prompts an activity tracking and mood reflection task and then offers a sleep meditation (Figure 1). Additional therapeutic content such as cognitive restructuring, thought recording, journaling, anxiety management, etc is delivered in a weekly cadence over 8 weeks. Each week, users receive a push notification that directs them to a report that details their app engagement thus far. Accessing the report also launches a congratulatory message for their engagement and unlocks access to new therapeutic tools as a reward for their engagement. Users can also independently navigate to therapeutic tools at any time, which are organized into “tool packs.” As a safety feature, the app also includes an “SOS” safety plan feature, which lists crisis hotlines and allows users to enter and store self-reported sources of hope and “safe people or places” to reference in moments of crisis.

Participants answered sociodemographic questions via an electronic REDCap questionnaire for descriptive purposes. Usability tests were conducted and recorded using audio and visual conferencing via a secure Zoom (Zoom Video Communications, Inc.) platform, each lasting approximately 45‐60 minutes. After participants downloaded Wysa for Chronic Pain to their mobile device, they were asked to complete the app’s onboarding process and complete several tasks within the app, all while performing “think aloud” testing, during which they described their thoughts and experiences in real-time (Multimedia Appendix 1: interview guide). During this process, the facilitator also took note of participants’ hesitations and inaccuracies in navigating the app, and the facilitator requested feedback on potential JITAIs that could be incorporated into the app. JITAI options and their details were iteratively refined throughout the usability testing process via a rapid iterative testing process. Based on feedback from the first 21 participants, mock-ups of revised app screens and workflows were created and shared with the next 13 participants using Figma (Dev Mode) and Twine (Chris Klimas) software. Additional minor refinements were made to the mock-ups for review by the final 8 participants. However, the evaluation tasks and the app’s core functionalities remained consistent for all participants.

The Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies (FRAME-IS) was used to identify and report potential usability-related adaptations needed for our target user population (Table 1) (Figure 2) [49]. FRAME-IS builds from the Framework for Reporting Adaptations and Modifications-Enhanced and is organized into modules to systematically document modifications to implementation strategies [50]. We elected to use the FRAME-IS rather than the Framework for Reporting Adaptations and Modifications-Enhanced because our goal was to adapt the delivery of therapeutic content within the digital health intervention (ie, usability and engagement optimizations as an implementation strategy), rather than adapt the core therapeutic content itself (ie, cognitive behavioral therapy, mindfulness exercises, etc as the evidence-based intervention).

Specifically for developing usability-related JITAIs, we applied Nahum-Shani’s pragmatic framework for JITAI development and the Behavioral Integration Technology (BIT) model [52,53]. Nahum-Shani’s pragmatic framework facilitated the systematic identification of opportunities for intervention using a JITAI [52]. That is, the framework was used to operationalize intended temporal progression of events through the app, which led to identification of promising opportunities for intervention with a JITAI. The BIT model facilitates effective translation of behavioral interventions into digital delivery systems. Specifically, we used the BIT model to design the JITAIs’ instantiation elements with intentional incorporation of behavior change strategies, implementation science principles, and user interface design principles that appeal to our target population [54].

Using a framework-informed hybrid inductive-deductive content analysis approach [55], iterative versions of general usability adaptations and usability-related JITAIs were developed and refined throughout the usability testing process based on feedback from participants and mapping to the analysis frameworks. As usability tests were completed, audio recordings of the usability tests were professionally transcribed (Rev.com), cleaned, and chunked (CYG). Three members of the study team (ALC, CYG, AM) collaboratively conducted open coding of the first 5 transcripts using an inductive, data-driven approach to identify concepts emerging from the data. Through discussion and consensus, they developed an initial codebook, which was then applied independently to pilot code the next 3 transcripts. Interrater reliability for these three transcripts was approximately 90%, with fewer than 5 coding discrepancies across coders. Discrepancies were resolved through consensus, and the codebook was refined accordingly. As usability tests were conducted with additional participants, the finalized codebook was applied using an independent open coding approach to these additional transcripts, each of which was coded by a single team member (CYG or AM), with periodic cross-checks between coders to ensure coding consistency. Coding was managed using NVivo 15 (Lumivero). Three team members (ALC, CYG, and AM) collaboratively and iteratively grouped related codes into preliminary themes that reflected participants’ usability needs and suggested app adaptations, including JITAIs. This process was continued until analysis of additional transcripts no longer yielded new themes. Then, we conducted a deductive analysis by examining the inductively derived themes in relation to the FRAME-IS framework, Nahum-Shani’s pragmatic framework for JITAI development, and the BIT model. The 3 team members (ALC, CYG, and AM) reviewed and discussed the themes to determine their alignment with these guiding frameworks, refine interpretation where appropriate, and translate findings into a final set of proposed general usability adaptations and usability-related JITAIs.

Usability testing was facilitated by a senior female research coordinator who has clinical research experience working with older adults and in the fields of mental health and orthopedics (AM). Participants’ only contact with the facilitator was related to study recruitment, consent, and participation. The other 2 research team members who contributed to codebook development included a female physician scientist with a clinical background in physical medicine and rehabilitation (ALC), as well as a female physician trainee in physical medicine and rehabilitation residency (CYG). The clinical and musculoskeletal background of these team members may have influenced data collection and interpretation such that physical barriers and related adaptations were disproportionately considered over solutions that considered mental health and systems-level factors. We worked to balance this potential bias by incorporating input from a clinician scientist with mental health clinical expertise (SH) and a health information technology expert (JA).

Informal member checking was performed by the facilitator (AM), who paraphrased and restated participants’ feedback during the usability testing process to confirm that participants’ intent and insights were accurately understood. Additionally, preliminary themes that emerged from early interviews were shared with later participants to assess for perceived validity. Participants generally affirmed the face validity of these themes.

In this study, we identified adaptations to an existing mental health app that better meet the stated usability preferences and needs of nondigitally native adults who have chronic pain and coexisting symptoms of depression or anxiety. Study participants specifically identified several opportunities to optimize their user experience by ensuring that the digital health intervention clearly describes how all its features are intended to be used and by minimizing navigation burden. For instance, users proposed general usability adaptations for a more robust orientation experience during app onboarding and for ongoing personalized engagement prompts using usability-related JITAIs.

Based on prior literature and feedback from participants in this study [15,58], when identifying decision rules for our proposed JITAIs, we aimed to balance the use of passively collected app engagement data and user-directed triggers in order to respect users’ dual desires for reduced navigation burden and also maintenance of autonomy and privacy. In related work [12], the majority of users in this target population expressed hesitation about allowing a digital mental health app to collect and use other data, such as physical location that is passively collected from their mobile device or other wearable sensors. Furthermore, older adults and those with multimorbidity are less likely to continuously carry a mobile device or use a wearable sensor, which would limit the impact of JITAIs that rely on datapoints from these sources [59-62]. As a result, the JITAIs developed in this study were intentionally designed to operate using only engagement-related and user-reported data that is collected within the app itself.

From a practical standpoint, the proposed JITAIs identified in this study represent general concepts that were perceived favorably by users. Determining the effectiveness of the proposed JITAIs and identifying ideal specific cut-points for each JITAI’s tailoring variables (eg, number of hours or days of no app engagement before delivering an engagement-related JITAI) requires additional investigation. Although preliminary cutoff points for proposed JITAIs can be developed based on qualitative evidence from a usability study such as this, recommended best practices suggest follow-up investigation using complex analytical techniques to refine evidence-based (and potentially adaptive) cutoff points [15]. Implementation of potential JITAIs also depends on feasibility within the digital health platform, as well as confirmation that the final decision rules and cutoff points of the JITAIs do not conflict with one another or with other elements of the intervention’s features or user experience. We plan to refine the details of these JITAIs using a series of microrandomized trials and additional qualitative user feedback [41].

In addition to leveraging existing evidence-based frameworks, we also leveraged an academic-industry partnership to expedite the incorporation of evidence-based principles into a digital health intervention with potential for rapid dissemination to the general public [63]. Specifically, the commercial app company facilitated participant access to the intervention and iterative mock-ups for the usability testing. The academic researchers led the usability testing itself and applied evidence-based frameworks to guide data collection and analysis. After the identified adaptations are fully built and subsequently tested for effectiveness, they can rapidly be rolled out to several thousand daily users in a much faster time scale than would be possible if an academic group developed a digital health intervention de novo. Our process to identify implementable usability-related adaptations of an existing mental health app can serve as a proof-of-concept method for relatively rapid refinement of other existing digital health interventions in order to better serve specific target populations, especially for apps that already have robust evidence-based content but could benefit from adaptations for specific subpopulations.

A major strength of this study is the integrated framework-based approach to identify usability-related opportunities for a target user population with unique needs. This approach can serve as a model for other studies, and the usability-related principles that were identified in this study are broadly generalizable for other digital health interventions that are intended for nondigitally native adults with multimorbidity. The robust sample size is also a strength relative to other digital health usability studies [64,65].

Some limitations are inherent to the study design used. Observer bias (ie, the Hawthorne effect) may have resulted in participants reporting more confidence or demonstrating greater tenacity with technical features than they otherwise would in a real-world setting. Additionally, the iterative nature of our usability testing resulted in some heterogeneity in participant experiences. While participants were not exposed to identical interfaces, this approach is consistent with user-centered and formative usability evaluation methods, in which iterative refinement is prioritized to rapidly identify and address usability barriers [66]. Furthermore, details of some of the usability-related adaptations and JITAIs that were identified in this study are predominantly applicable to the specific mental health app that was tested. Our proposed adaptations also reflect participants’ stated preferences, but quantitative assessment and data-driven refinement was outside the scope of this study. Finally, user preferences among nondigitally native adults may evolve over time as the population’s overall experience with technology continues to grow.

Given the demographics of our sample, our study findings are likely most generalizable to middle-aged adults in the United States who are relatively digitally proficient and already use the internet routinely throughout the day. As a result, findings from this study may not fully capture the usability needs and other barriers faced by older adults with low digital literacy, which remains an important and underserved segment of the target user population. Interpretation of generalizability is also limited by the self-selected nature of the convenience sample and the lack of a standardized cognitive screening instrument, especially since the prevalence of mild cognitive impairment increases with age.

Identifying implementable opportunities to improve usability and prompt meaningful engagement at relevant timepoints holds promise to improve the effectiveness of digital health interventions because intervention attrition is a well-known challenge that is suspected to reduce the effectiveness of digital health interventions on a population level [67-70]. In this study, we used an integrated framework-based approach and user feedback to identify usability-related optimizations to an existing mental health app that are preferred among relatively digitally proficient but nondigitally native adults who have chronic pain and coexisting symptoms of depression or anxiety. Users identified general usability-related adaptations and usability-related JITAIs which were appealing for their potential to (1) provide clarity with regard to how to navigate the digital health intervention, and (2) reduce the user’s burden of independent navigation within the intervention. These usability-related refinement opportunities are generalizable to other digital health interventions intended for older users who are likely to be managing multimorbidity, are not digital natives, and are more likely to have physical or cognitive impairments than younger users (eg, medication reminder apps, daily safety monitoring apps, etc). The integrated framework-based approach taken in this study can also serve as a model for refinement of digital health interventions with other target populations, as well.