Workload, Usability, and Engagement with a Mobile App Supporting Video Observation of Methadone Take-Home Dosing: Usability Study

Introduction

Opioid use disorder (OUD) remains a major cause of mortality in the United States [1]. Methadone is 1 of 3 OUD pharmacotherapies approved by the US Food and Drug Administration but requires frequent in-person observed dosing (ie, direct observed therapy [DOT]) at a federally certified opioid treatment program (OTP) to mitigate the risks of medication diversion and overdose. The requirement of DOT can impose barriers for patients and limit access to treatment [2].

Mobile health (mHealth) technology has the potential to help reduce barriers to methadone treatment [3]. For example, smartphones allow patients to video-record themselves taking methadone at home or send messages to clinical providers, which can reduce the need for frequent visits to an OTP for DOT. However, in-person DOT remains the standard by regulation in OTPs [4].

In response to concerns about respiratory illness transmission during the COVID-19 pandemic, a large OTP agency with 3 separate sites in Washington state conducted a pilot program between April and August 2020 aimed at reducing the need for in-person DOT and doing remote screenings of COVID-19 symptoms. During the pilot, a subset of patients was invited to use the Emocha mHealth app to facilitate video-based DOT for all methadone take-home doses along with COVID-19 symptom screening completed with each video DOT submission.

In a study (Hallgren et al [5]) describing the clinical pilot, we showed that patient adherence to video DOT varied, but on average, video DOT significantly increased the number of days of observed methadone dosing and most patients received increased methadone take-home dosing privileges due to their ability to demonstrate treatment stability. However, the direct usability of the video DOT app has yet to be tested with patients or counselors in methadone treatment settings.

For this study, patients and counselors who participated in the pilot program were invited to participate in an evaluation of the app’s overall usability. We hypothesized that the mobile app would have favorable workload, usability, and engagement for patients with OUD receiving methadone treatment, including for patients who had higher versus lower adherence to the app during the original pilot program. We also hypothesized that counselors would report favorable workload, usability, and engagement for the provider-facing app portal. 

Methods

Study Design and Sample

Patients and counselors who participated in the original pilot program and were still receiving care or employed by the methadone treatment program were invited to participate in the usability study between May and August 2021. Patients and counselors were invited using phone calls, letters, and flyers distributed at the OTP. Recruited counselors were also encouraged to refer participants from the pilot to take part in this usability study. Efforts were made to recruit patients who in the original clinical pilot had low adherence (less than 18 video uploads), medium adherence (18-45 video uploads), and high adherence (more than 45 video uploads), defined by terciles of video uploads. Additional information describing the original pilot program and outcomes was reported by Hallgren et al [5].

Testing Procedures

Participants completed a single usability session conducted 1-on-1 with a research coordinator in a private setting following a standardized protocol. After providing informed consent, participants completed a demographic questionnaire. The research assistant (RA) administered usability testing tasks and questionnaires and recorded data into REDCap. To complete usability testing tasks, patients either used the mobile app on a study phone or downloaded the app to their phone.

In the first session of the usability study, our RA provided verbal instructions and prompts as participants engaged in each of the tasks. This approach was chosen because, unlike a self-help app where users typically interact with the app without guidance, the Emocha app is designed for users who receive instructions from health care staff on how to complete specific tasks. Therefore, providing instructions during the usability testing accurately reflects the intended user experience and was seen as the most appropriate methodology for this study. One illustrative example of the scripts used by our RAs during the study is as follows: “Please open the mobile app and log on using the provided username and password.”

Patient participants were asked to complete 5 tasks that were determined by the study team to be the most important for successful video DOT: logging into the account, completing a COVID-19 symptom screener, uploading a video of themselves simulating methadone ingestion, sending and checking messages to a counselor, and accessing and reviewing a calendar showing methadone adherence. Counselors completed usability testing tasks on the provider-facing web portal using a study computer if the visit was in person or through their own work or personal computer if the visit was conducted remotely through Zoom. Counselor participants were asked to complete 5 tasks, that is, add a new patient, review 1 patient video, change a patient’s video regimen time and number of uploads, send and check messages, and check the patient “adherence calendar.”

The research coordinator timed each task and observed whether it was completed successfully. Participants provided NASA Task Load Index (NASA-TLX) ratings after each activity. After all activities were completed, participants completed the System Usability Scale (SUS) and User Engagement Scale-Short Form (UES-SF), described below.

Ethics Approval

All procedures were approved by the University of Washington Institutional Review Board (review number STUDY00011142).

Measures

NASA-TLX Measure

The NASA-TLX is a validated measure [6] of the cognitive workload required to complete a task. Participants self-report the mental demand, physical demand, temporal demand, performance, effort, and frustration associated with each of the 5 activities completed during the usability testing session on a visual analog scale of 0-100. An overall task load index was computed as the unweighted mean rating across all 5 activities [7]. We derived the following cutoffs to interpret mean workload: <33 for low workload, 33-66 for moderate workload, and >66 for high workload. These cutoffs were informed by Patel et al [8], who reviewed workload ratings for 21 electronic medication adherence apps and found an average workload of 50 (SD 26) with some of the least workload-heavy products having mean ratings of around 29.

SUS Measure

The SUS is a validated self-report usability measure [9]. It has 10 statements (5 positively framed and 5 negatively framed) that are rated on a 5-point scale completed at the end of the testing session. Total scores were calculated following standard instructions [10] to produce a total score ranging from 0 to 100, with higher scores indicating greater usability. Similar to our NASA-TLX scores, we interpreted mean scores <33 as low usability, 33-77 as moderate usability, and >77 as high usability. These cutoffs were informed by previous studies showing that the most usable medication adherence products had mean SUS scores of about 78 (SD 15) and the least usable medication adherence products had mean scores of around 28 (SD 21) [8].

UES-SF Measure

User engagement reflects the depth of cognitive, temporal, affective, and behavioral investment when interacting with a digital system [11] and was measured using the UES-SF [12]. The UES-SF is a 12-item self-report measure with 4 dimensions reflecting focused attention, perceived usability, aesthetic appeal, and reward factor; the latter subscale combines a felt sense of novelty, involvement, and endurability experienced while interacting with the digital system. Each question is answered on a 5-point rating scale. Following recommendations by O’Brien [11] and O’Brien et al [12], we calculated mean scores for each subscale and an overall engagement score reflecting the mean rating of all 12 items (negative engagement items were reverse coded). For this analysis, an average score higher than 3.5 would indicate high engagement.

Analytic Approach

Descriptive statistics characterized the patient sample. Workload, usability, and engagement measures were analyzed descriptively by computing means and 95% CIs of composite indices within the patient and counselor cohorts. Additional descriptive analyses were performed within patient subgroups who, during the original clinical pilot, had video DOT adherence that was considered low to moderate (n=8 patients) and high (n=4 patients) during the first 60 days from enrollment.

Results

Description of Sample

The study recruited and enrolled 12 of the 60 patients who participated in the clinical pilot (2=low adherence, 6=medium adherence, and 4=high adherence) and 3 of the 5 counselors. Table 1 describes the patient participants in the usability study. Demographics for counselors are not reported due to the small number of those participants. On average, patients were in their late 40s and most were male and White.

Task Completion

All 5 activities were successfully completed by all participants, with the exception that 1 patient did not successfully log into the app. It took an average 1.6 minutes for patients to complete each of the 5 activities. The most time-consuming activity was logging into the account, which took a mean of 3 minutes; however, this mean was greatly affected by 2 outlier participants who took 9.4 and 13.8 minutes to complete the task. Of the 10 remaining participants, 8 completed the task in less than 2 minutes, and 2 completed the task in 2-2.5 minutes. The RA observed that some participants took longer to complete the login task because of problems not directly related to the software. For example, 1 patient participant engaged in conversation while attempting to log in, which prolonged the process. Another participant entered an incorrect test password, resulting in failed login attempts. In another instance, a slow-performing phone impacted and slowed the process and appeared to create login failures. The second most time-demanding activity was sending and checking messages (1.9 minutes on average). For clinicians, it took an average 1.4 minutes to complete each of the 5 activities. The most time-demanding activities were changing a patient’s video regimen time and number of uploads and reviewing 1 patient video (both 2.2 minutes on average).

Workload (NASA-TLX)

Overall, patients and counselors reported low mental, physical, and temporal demands, successful performance, low effort, and low frustration associated with activities. However, counselors reported somewhat higher demands across all categories and activities. Results for the 2 video DOT adherence subgroups and counselors are presented in Tables 2 and 3.

Usability (SUS)

Mean SUS scores reflected high usability for patients (mean 85, SD 9.5). Usability was also high for the patient subgroups with low to moderate video DOT adherence (mean 87.5, SD 7.9) and high adherence (mean 79.3, SD 11.2; Table 4), a nominal difference that was not statistically significant (mean difference 8.5, 95% CI –3.8 to 20.8). In contrast, usability ratings for counselors were considerably lower (mean 43.3, SD 17.7).

User engagement (UES-SF)

User engagement was high for patients (mean 3.8, SD 1.1). User engagement was high for the low to moderate video DOT adherence subgroup (mean 3.9, SD 1.2), but it was lower for the high adherence subgroup (mean 2.8, SD 1.1), a nominal difference that was not statistically significant (mean difference 1.1, 95% CI –0.5 to 2.7). Results for specific domain categories are described in Table 5. For counselors, engagement was lower (mean 2.8, SD 0.6), particularly for the reward and perceived usability domains.

Discussion

Overview

Methadone is a life-saving medication for patients with OUD, but requirements for in-person DOT can impose significant barriers to treatment. This study found that an mHealth platform that facilitates video-based DOT and COVID-19 symptom screening through smartphones required low workload and had high usability and engagement for patients with OUD receiving methadone and required low workload and had moderate usability and engagement for methadone treatment counselors. However, counselors scored lower across all instruments. Results indicated that the 5 most critical functions of the app could almost always be completed by patients and counselors and that these tasks were associated with low cognitive workload, high usability, and high user engagement, including for patients with low to moderate adherence and high adherence in the original pilot.

Results suggest that video DOT can be usable for patients with OUD in methadone treatment. The strong performance observed for the study cohort, including in patients with low to moderate adherence in the original pilot, suggests that usability was unlikely to be a significant barrier to adherence with video DOT and that other barriers may have contributed more to variability in video DOT adherence. Contextual factors influencing experiences using the app are currently being explored in a separate qualitative study.

Further investigation is necessary to determine the reasons behind counselors’ lower ratings of the mHealth platform’s usability and engagement compared to patients. It is worth noting that the counselor-facing portal had different features than the patient-facing app. For example, the tasks of counselors were different from patients, as patients submitted videos that counselors then reviewed and approved. We speculate that counselor usability ratings could have been impacted by software and clinical workflow issues. These factors need further investigation, especially given the small sample size in our study. However, possible solutions to address these issues could include providing additional training and support to counselors to enable them to feel ownership, ease, and mastery of their role, such as adding more advanced features or customization options to the platform and conducting more rigorous testing and evaluation of the counselor-facing portal to identify and address any specific usability issues. Future work could also examine how to improve the integration of video DOT with the counselors’ clinic routines and existing workloads, including by identifying ways to minimize the potential impacts of such systems on their clinical routines.

Our study has limitations, such as the small sample size, especially for counselors. With 2 of 5 counselors missing, the findings might be unrepresentative and biased. Although descriptive analyses can be conducted with small sample sizes, the precision of results is limited, and we could not perform subgroup analyses to evaluate usability within important patient subgroups (eg, patients experiencing homelessness or higher-severity OUD). There may have been sampling bias, as we were only able to recruit patients and counselors who were in the original pilot program and were still in the clinic over 1 year later. We also recruited only 2 patients with low adherence in the original clinical pilot, which may introduce a bias toward more favorable results, as one would hypothesize that patients with low adherence might be more likely to experience problems with usability (however, several usability ratings were nominally higher for participants with low to medium adherence compared to patients with high adherence). Our study also has strengths, including its focus on analyzing a novel method for DOT for methadone and usability testing with people who may often be overlooked in technology development efforts.

Conclusion

Little knowledge exists on the usability of mHealth apps for patients and counselors in methadone treatment. This study narrows the knowledge gap by providing information on workload, usability, and engagement with an mHealth app delivered through smartphones for video observation of methadone home dosing and COVID-19 symptom screening. The study demonstrated that a mHealth app to facilitate video-based DOT of methadone was unlikely to create a heavy workload for patients and counselors. Furthermore, there were no trends to suggest that adherence to the app in the original clinical pilot was related to workload, usability, or engagement, indicating that factors unrelated to usability may have impacted adherence when the app was used during the clinical pilot. Although the app was well received by patients, the study highlights opportunities to improve and further investigate usability for counselors, perhaps by improving training or care integration.