The study has been approved by the Institutional Review Board of the VA Greater Los Angeles (1615834-20).

This study was conducted as part of research developing a mobile sensing informatics intervention and conducting pilot use of the intervention in a population with SMI. The protocol, including the details of the methods for this study, has been previously published [16]. Patient focus groups and usability tests were conducted to determine the acceptability and usability of the Mobile Sensing app for self-monitoring while clinician interviews determined the acceptability of the mobile sensing data to improve clinical care. Data were collected during the preimplementation user-centered design phase of the passive mobile sensing study (October 2020 to June 2021). This focused on usability and perceptions regarding the mobile app and informed modifications to the product during phase 1 of the mobile sensing study design (user-centered design phase) using patient focus groups and usability tests to inform mobile app modifications for phase 2, the mobile sensing phase of the study [16]. Results from this study contribute to the field of passive mobile sensing technologies and self-tracking in patients with SMI to improve clinical care and patient outcomes.

Our team developed a functional mobile app for Android cell phones that passively tracks behavior in patients with SMI. Using mobile sensors, phone use, and phone communication, the app collects data that are relevant to 3 behavioral domains: sleep, sociability, and activity. Phone sensors (eg, accelerometer sensors and ambient light sensors) collect data on location, movement, sound, and light; phone usage transmits data related to apps used and screen on-time; and phone communication transmits data on the number of calls and SMS text messages placed (not content). These input data are used to develop individualized estimates of the 3 behavioral domains previously listed. Figure 2 shows the digital dashboards of the app for the 3 behavioral domains by type, intensity, and over time.

Qualitative data were collected from patients in treatment for SMI and clinicians providing mental health care (eg, psychiatrists, psychologists, nurses, and social workers) at 1 Veterans Administration (VA) medical facility and is summarized in Table 1. Qualitative researchers assisted in data collection instrument design, development of interview guides, transcript verification, and data coding and analysis. Interview and focus group guides were written using HITAM as a framework. The method of administering minimized biases by asking open-ended questions before more direct probes.

Semistructured interviews with 16 clinicians were conducted between December 2020 to March 2021. Interviews lasted approximately 30 minutes, were audio recorded, and transcribed with participant consent. Interviews first aimed to assess clinicians’ general interest and perspectives on using the Mobile Sensing app, its web-based clinical dashboard, and its data (eg, passive data monitoring patients’ sleep, activity, and sociability) in their clinical practice. Then, feasibility, acceptability, facilitators, barriers, and suggestions for patient self-care and for clinician use in clinical practice were discussed.

Three 45-minute focus groups of 4-6 patients (n=17 in total) in treatment for SMI were conducted between February and April 2021. Participants were recruited through in-person and web-based study recruitment presentations made at the Psychosocial Rehabilitation and Recovery Center and the Domiciliary Residential Rehabilitation Center, and through study flyers posted in the facility’s mental health clinic. Participants were paid US $20. Focus group discussions consisted of showing pictures of the Mobile Sensing app prototype (see Figure 2) and describing what dashboards, charts, and graphs showed them and their clinicians. They generated data on patients’ preferences, interests, and critical input to inform the intervention development.

Usability tests were conducted with patients in treatment for SMI between April and June 2021 (n=8). Participants were recruited from our focus groups. Each usability test lasted approximately 30 minutes and participants were paid US $20. Patients participated in usability tests individually using an Android phone provided by a study team member; tests concentrated on downloading, opening, reviewing the dashboard, and closing the app to track patient feedback to inform a more user-friendly app.

A team of 3 analysts conducted a directed content analysis [29] using a rapid qualitative analytic approach [30-32]. We began with a priori coding categories based on interview guides developed using HITAM measures. Emergent categories captured additional content relevant to patients with SMI and the VA health care system. Consistent with a rapid qualitative approach [30], team members reviewed interview notes, summarized, and validated information by category into tables to capture relevant content sorted by patients and clinicians. Information was synthesized across individual summaries, using constant comparison, to understand patient and clinician perceptions regarding the utility and acceptability of the smartphone app. This paper focuses on results regarding the utility and acceptability of a hypothetical mobile app that uses mobile sensing and a clinical dashboard.

All clinicians who completed an interview (n=16) provided mental health care to patients with SMI. Further, 7 psychiatrists and 1 nurse provided clinical care, 3 psychologists provided psychosocial casework, and 5 social workers were case managers. Half of the clinicians worked primarily at a psychosocial rehabilitation and recovery center, while the others worked in a residential rehabilitation program or general medicine. Table 2 summarizes clinicians’ roles and the departments in which they work at the VA health care system.

Of the 21 patient participants recruited, 17 consented and participated in 1 of the 3 focus group discussions. All focus group participants (which includes the usability testers recruited from focus groups) were male; most were in-patients (n=13, 77%), and almost a quarter were outpatients (n=4, 23%). Demographically, focus group participants were diverse, comprised of a range of ages (median 45, range 21-66 years), an equal number of self-identifying non-Hispanic White and Black participants, 2 self-identifying Hispanic participants, and some who declined to respond to race or ethnic identity questions.

From focus groups with patients with SMI, we found that most reported that they continuously use apps on their smartphones for various purposes.

Most patients downloaded apps on their phones themselves; however, a few said that they had other people download apps for them or that their phones came with apps already downloaded. Participants reported that they used apps to access the following: music (eg, Spotify and iTunes), social media (eg, Instagram and YouTube), health (eg, MyHealtheVet, tracking steps, and managing cholesterol or weight), travel (eg, Waze), or fast-food restaurants (eg, McDonald’s and Taco Bell). Feedback about the Mobile Sensing app specifically (see Figure 3) was mostly positive. About half of patient participants stated that the Mobile Sensing app would be easier to use than most other apps and participants generally liked the idea of using the app for self-monitoring.

As the above quotations illustrate, most patients were interested in apps with information that would help them self-monitor and manage their illness. Some commented that the Mobile Sensing app would be useful for “keeping out of trouble” as the app would encourage them to self-regulate. They said it would be easy to download onto their smartphones and to access and use the dashboards. Some found the app “innovative,” easy to read, and useful for self-monitoring their sleep.

Patient perspectives about clinicians viewing dashboards and receiving alerts were mixed, but mostly positive (Textbox 1).

A few participants reported that the passive data would help clinicians gain better insights about their individual health status. A few also stated that the passive data would be more accurate than their self-reports. The advantages to using the app were that patients did not need to remember to turn it on, that it reports data automatically, and that it would be good to be contacted by a clinician if they noted significant changes in their behaviors. Concerns about passive monitoring included privacy factors, not understanding how it works or differentiates activities (sleep, walking, and driving) and where data goes and to whom, particularly the government “seeing” your social life. Other patient concerns were remembering to look at it and turning it off accidentally. Despite their concerns, participants stated that they would use the app if it were available to them as long as it did not drain their phone battery or use up data (behavioral intention).

Findings from individual usability tests with individual testers to inform user-centered app design echo feedback we received during focus group discussions. Downloading and opening the Mobile Sensing app proved easy for smartphone users, with some expressing concerns about potential problems for older patients. Overall, patients with SMI did not have problems downloading, opening, closing, or understanding the app dashboards that the following quote exemplifies.

Most reported that graphs and numbers were clear and easy to understand, while a few were not initially sure what D (day), W (week), or M (month) signified. Testers made 2 key suggestions for improving the usability of the app, posting tutorials or demonstrations on the VA store web page, and optimizing font sizes for graphs and other graph components.

From interviews with clinicians, we found that most had no experience using mobile apps for mental health in their clinical practice and that half were unaware if their patients with SMI used mental health applications. According to clinicians, the idea of using mobile apps to passively monitor the health trends of patients with SMI was viewed as potentially advantageous but with notable caveats:

Some clinicians felt that mobile apps were the “new direction of psychiatry,” that passive data were more feasible (eg, reliable) than active (eg, patient-reported) data, and that using the Mobile Sensing app may be a better way to support the SMI population by providing an added layer of clinical care in conjunction with normal care by helping them know more about patients’ health status, trends over time, and receiving alerts. However, clinicians expressed concerns about the clinical usefulness of the data and the need to tailor or individualize data to know if a patient was experiencing worsening symptoms.

About half of clinicians (n=7) anticipated challenges in “selling” the idea to their patients with SMI due to projected SMI patients’ paranoia or older patients’ reluctance to use technology. Clinicians perceived advantages of Mobile Sensing for patients were that patients did not have to actively participate, that they could “see evidence” to better understand their mental health, and that the app may motivate them to seek help sooner. Clinicians perceived disadvantages for patients were paranoia, not having or losing their smartphones, and potential technological limitations. A few clinicians wanted more information on how the passive data played out in real-life situations.

When asked about the usefulness of passive data, clinicians said that it may give them a better sense of what is going on in patients’ lives, since self-reports are subjective and hampered by recall bias.

Over half of clinicians interviewed said that, if patients used it and they could establish a baseline, they would very likely review data from the app to monitor patients’ mental health trends over time. They felt that the app could be used as a reporting tool, similar to exercise apps that track steps, and that since the dashboard data seemed visually easy to review it may open up time during regular appointments to cover more with patients. Others felt the data would be useful for knowing when symptoms were ramping up and the need for medication adjustments.

Most clinicians found receiving alerts for worsening mental health symptoms useful, but with caveats.

About half of the clinicians said that alerts would be useful for managing large caseloads and detecting major behavioral trends. There were a few concerns about receiving them, which focused on the responsibilities of responding to them outside of work hours and on weekends. Most felt that receiving alerts by secure email via electronic health records (Computerized Patient Record System, CPRS; GTI INFOTEL) or having the dashboard integrated into Computerized Patient Record System (CPRS) would work best so that they could respond appropriately, either by phoning the patient or scheduling an appointment through VA Video Connect (VA) software within 24 hours.

Barriers to uptake of the Mobile Sensing app that clinicians discussed focused primarily on potential workload issues and alert fatigue.

All clinicians discussed workflow and that monitoring alerts and follow-up time may add uncompensated work. Clinicians suggested having a second person or team to respond to alerts during weekends, evenings, and time off as well as automatic connections to crisis lines with VA in severe situations. They also discussed the potential cultural shift for patients, being treated based on data that they have not self-reported and the “big brotherish” nature of passive data collection. Despite these barriers, most clinicians said that they would recommend the Mobile Sensing app to their patients (attitudes and behavioral intention).

This study contributes to the literature regarding health IT acceptability among patients with SMI and mental health clinicians. Framed by the HITAM model, findings from our study indicated that patients and clinicians were receptive to remote, passive monitoring using smartphone technology to rapidly detect worsening mental health symptoms. Patients with SMI appeared interested in self-monitoring and having alerts sent to their mental health clinicians (health concerns and perceived threats). They found the app easy to navigate and dashboard graphics easy to understand (perceived usefulness and ease of use). Clearly, our findings showed that clinicians lacked experience using apps in their clinical practice and using health IT in clinical care (health concerns and perceived threats). However, both patients and clinicians recognized the potential advantages of passive monitoring and receiving alerts for worsening symptoms (HIT reliability and perceived usefulness). Patients were concerned with privacy factors and data or battery issues, while clinicians were apprehensive about workload and alert fatigue (HIT self-efficacy and perceived ease of use). Despite these trepidations, most patients and clinicians stated that they would use the Mobile Sensing app if or when it was available (behavioral intention).

Our findings suggest that the quality of the app’s output, its reliability, and other factors, such as the health care environment and clinician experience using mobile technologies, are all significant considerations prior to implementation. While the US Department of Veterans Affairs offers a wide range of mental health–related apps [33], implementing a mobile digital tool for use by clinicians and patients is difficult [22,34,35]. It requires buy-in at the organizational and end user (clinician and patient) levels to achieve optimal outcomes [6,23,36,37]. While we found that almost all of the clinicians we interviewed had no experience using mobile technologies or health IT in their clinical practice at the time that they were interviewed, limited access to mental health services during COVID-19 facilitated the rapid development of digital clinics and mobile apps, affording clinicians exposure to and success with digital health [23,34,35,38]. Similar to these recent digital health studies, we found that it is critical to rethink the clinical workflow to successfully implement and integrate an app using mobile sensing. Additionally, due to the challenges of designing mobile technology for use by the SMI population [21], we integrated graphic design style and tutorial suggestions gleaned from patient focus groups and usability tests into the development of the Mobile Sensing app. Potential facilitation strategies for implementation could include using digital navigators for patients and educating frontline staff for clinical deployment.

Successful adoption of Mobile Sensing ultimately depends on the actual end users of the data, in this case, both patients with SMI and clinicians. The VA is moving toward a hybrid environment, especially since the pandemic. A hybrid environment drives app-based approaches to mental health care and creates challenges for recruiting patients. All these factors indicate the need for a plan to rollout and implement the project in a feasible way. The benefits of the Mobile Sensing app for patients include its function as a self-management support tool and an alert notification for behavioral changes. For clinicians, it provides a care coordination support tool that notifies them of changes in patients’ behaviors. A clinician will be able to contact a patient in a timely manner and alert the health care organization that someone needs a check-in for their mental health or other chronic conditions. However, clinicians also expressed reservations about time and workflow adjustments that using the Mobile Sensing dashboard may require of them. Despite these reservations, clinicians also acknowledge the assessment and treatment strategies being developed and deployed for treating mental health [6,39]

This work is limited by its restriction to 1 VA site and its generalizability to other populations (eg, female veterans and those outside the VA) and health care systems. The COVID-19 pandemic created challenges in conducting this research, leading to innovative data collection methods such as hybrid focus groups (ie, web-based and in-person) conducted outdoors, and may have limited our sample size and sample diversity. Alternative strategies may be needed for the most vulnerable patients with SMI such as low socioeconomic status populations with SMI who may not have consistent or any access to a smartphone.

This paper reported on phase 1 of the Mobile Sensing Study Design, User-Centered Design Phase. Recruitment and enrollment for the mobile sensing intervention phase (phase 2) began in October 2021 with the goal of recruiting 125 patients and is expected to conclude in July 2023. Qualitative and quantitative assessments during and after deployment of this phase will measure patient experiences as outcomes. Mobile phone sensor and utilization data will be used to develop individualized estimates of sociability, activity, and sleep that will also be measured through weekly interviews. Various machine learning algorithms will be used to build, train, and select prediction models for each patient’s behavioral assessment domains, and evaluated for predictive performance and cross-validation. Postsensing phase interviews will assess how to engage patients and reflect on findings, implementation issues, and resources needed for sustaining and incorporating mobile sensing with the Mobile Sensing app into routine practice.

Many people with SMI would like to have mobile informatics tools that can support their illness and recovery. Similar to other populations, such as older adults or people experiencing homelessness, this population presents some challenges to adoption and implementation. HITAM provides a useful lens with which to analyze the acceptability and usability of mental health mobile apps. Health care organizations will need to provide resources to address these and support successful illness management among these populations. Clinicians are also supportive of technological approaches, with adapting to using informatics data in their workflow as the primary challenge. Despite clear challenges using technology-based assessments like mobile sensing, technological developments are exciting and increasingly designed to be acceptable to patients. The research development–clinical deployment gap will have to be addressed by health care systems, similar to the case for computerized cognitive training. It will be necessary to ensure clinicians operate at the top of their skill set and that they are not overwhelmed by administrative tasks, data summarization, or reviewing data that does not indicate a need for their intervention.