Currently, at least one in three people will develop type 2 diabetes mellitus (T2DM) in his or her lifetime [1]. People with diabetes are at higher risk of critical health complications including kidney failure, heart disease, and stroke [1]. More than 20% of health care spending in the United States goes toward people with a diagnosis of diabetes [1]. Racial and ethnic minorities are more likely than non-Hispanic whites to have a diagnosis of T2DM [1] and, once diagnosed, have more diabetes-related complications [2], hospitalizations [3], and premature death [4].

People with T2DM can take medication, eat healthily, exercise, and test blood glucose levels to achieve optimal glycemic control [5] and, in turn, prevent diabetes complications [6] and premature mortality [7]. However, the initiation and maintenance of these self-care activities is challenging [8], and rates of self-care adherence are low among adults with T2DM [8,9]. Adherence rates are even lower among racial and ethnic minorities [10-12] and persons of low socioeconomic status (SES) [13] owing in part to financial difficulties and misconceptions about diabetes and self-care (eg, believing they do not have control over their diabetes, believing medication is not important) [8,14-16].

Mobile phone–based interventions using text messaging are a practical approach for improving medication adherence among low-SES, racial and ethnic minorities with T2DM. More than 90% of US adults own a mobile phone [17]. Although smartphones are used less among individuals with diabetes [18], lower income [19], and lower education [19], text messaging does not require a smartphone and is the most common activity among all mobile phone users, used equally across SES, race, and ethnicity strata [20,21].

Text messaging interventions are improving the self-care and glycemic control of adults with diabetes [22-24], but few have been tested with disadvantaged populations in the United States [25]. Two prior text messaging interventions [26,27] improved glycemic control in low-SES samples but not relative to a control group. A third text messaging intervention [28] improved glycemic control among a racially diverse sample but this sample had relatively high SES. Each of these interventions use text messages to address barriers to self-care, but none identifies and addresses each user’s unique barriers. Such barriers vary from person to person with T2DM [29-32], requiring a tailored user experience.

Tailoring text messages to a user’s unique adherence barriers can address issues most applicable to him or her, such as limited diabetes knowledge, negative beliefs about medication (eg, fear of side effects), or limited financial resources [16]. Although interventions cannot easily target a person’s SES, they can enhance one’s problem-solving ability to address financial barriers and other modifiable barriers [14,16]. We developed the MEssaging for Diabetes (MED) intervention that sends tailored text messages addressing user-specific barriers to adherence and text messages assessing adherence [33]. After 3 months of MED among disadvantaged adults with T2DM, users’ barriers were reduced and barrier reduction was associated with improved glycemic control [34]. Furthermore, MED users were highly engaged, responding to 84% of daily assessment messages, and engagement did not differ by sex, race, income, health literacy, or duration of diagnosed diabetes [35]. MED’s findings are consistent with reviews suggesting text messaging interventions with personally relevant, tailored content are more engaging [36] and effective [37] than those without tailored content.

Additionally, there is mixed evidence as to whether theory-based interventions are more effective than atheoretical interventions [38,39]; this is in part due to interventions not extensively applying theory and using theories that are inappropriate for behavior change [38]. The Information-Motivation-Behavioral skills (IMB) model suggests that adherence to a behavior depends on behavior-specific knowledge, personal and social motivation, and behavioral skills [40]. The IMB model is empirically validated among a wide range of diverse samples of adults with T2DM, including samples with low SES [41-43], and explains more than 40% of the variance in their medication adherence [43]. Text messaging interventions for T2DM self-care are rarely based on health behavior theory [24,44,45]. Of the few text messaging interventions tested among disadvantaged populations with T2DM, only 2 mention using a theory-driven approach [27,28]; however, the extent to which theory was applied in these interventions is unclear.

We developed Rapid Education/Encouragement And Communications for Health (REACH), a tailored, IMB model–based text messaging intervention. We performed 3 rounds of usability testing with adults with T2DM receiving care from Federally Qualified Health Centers (FQHCs) to identify and address any content and functionality issues before evaluating REACH’s effects on self-care and glycemic control.

REACH was developed with MEMOTEXT, an algorithmic communications and data management platform supporting personalized user outputs and inputs via short message service (SMS); interventions using this platform have been tested with diverse patient populations with different health conditions who found them acceptable, engaging, and whose adherence improved >30% [46,47]. We worked with MEMOTEXT to develop REACH based on our experience developing and testing MED [33-35].

An average of 12 participants experienced REACH each testing round, totaling 36 participants (Table 4). The average age of the participants was 52.4 (SD 9.5) years, 63% (22/35) were a racial or ethnic minority, 39% (14/36) had less than a high school degree or equivalent, and 67% (22/33) had an income less than US $35,000. The average HbA_1c level was 8.2% (SD 2.2%); 64% (23/36) of the participants had suboptimal glycemic control (HbA_1c≥7.0%). Across rounds, the most frequently reported barriers to medication adherence were forgetting to take doses (56%, 20/36 users reported this barrier with an average score of 5.2, SD 3.0, of 10), the high cost of medication (44%, 16/36 users; mean score 6.2, SD 2.7, of 10), and believing that taking medication is unpleasant (42%, 15/36 users; mean score 5.2, SD 2.6, of 10). The most commonly reported insulin-specific barrier was feeling embarrassed to take insulin in public (35%, 6/17 users who were prescribed insulin; mean score 4.5, SD 2.4, of 10).

All 36 participants completed an exit interview. Overall, participants said REACH was helpful and gave favorable feedback on each intervention element. Participants reported preferences and technical issues requiring iterative improvements between testing rounds. Below, we describe this feedback, our iterative changes by intervention element, followed by changes in our research processes.

Across rounds, participants rated the ease of understanding and helpfulness of the daily self-care text message, on average, 9.6 (SD 0.7) and 9.3 (SD 1.4) on a scale of 1-10, respectively. Participants appreciated that these messages were simple and without medical jargon. Participants said inspirational messages made them feel supported and not alone in living with diabetes and motivated them to take more initiative with self-care. Messages with self-care tips and information were helpful because they either provided a useful reminder or communicated something new.

Interviewer: Why did you read those messages? What made you want to?

Participant: They was [sic] helpful. Some things I didn't know [sic]. They helped me understand a lot of stuff because I didn't understand. [37-year-old, African American male]

Many participants valued reminders to care for their diabetes when they otherwise might not think about it:

I thought these messages were very helpful. I get so busy in the day that I don’t [even] take time to eat. And then when I get a text, [I realize] oh, wow, I need to do something. That really helps a lot. I wish I had somebody who did that for me all the time. [59-year-old, white male]

Despite the overall positive feedback about self-care messages, some participants had concerns. For example, a round 2 participant said a message provided a suggestion for remembering to take medications, without providing the steps for carrying out the suggestion. A few participants said some messages implied a problem when they did not have one (eg, “Struggling to take your diabetes medications every day? Talk to a loved one about what is getting in your way.”). To address such concerns, we revised all problematic text messages between rounds 2 and 3 to provide additional context and be less presumptuous (see Table 5 for examples of problematic and revised text messages).

Across rounds, participants responded to 96% of adherence assessment text messages and, on average, rated the helpfulness of these messages 9.1 (SD 2.1) out of 10. Assessment messages helped remind participants to take their medications and maintain their routine. One participant commented on these messages’ emotional and social support:

[The texts] keep you on task about what you should do...especially if someone doesn't have anybody around. You know it's kind of like having a family member around to remind you, “Hey, you should take your meds.” These [texts] make you feel like someone cares or is concerned about your health and makes sure you're taking care of yourself. So I think that's very helpful. [48-year-old, African American male]

Several participants with optimal glycemic control (HbA_1c <7%) said these messages were not particularly helpful because they routinely took their medication and rarely missed doses. Nonetheless, these participants endorsed the value of these messages for others recently diagnosed with diabetes and/or newly prescribed medication who do not have an established routine.

Round 1 participants complained about needing to respond to assessment messages several times before the system would accept their response. Upon viewing system-collected data, we learned that participants used different variations of “Yes” to respond (eg, “Yup” or “Yeah”). Therefore, we expanded the acceptable response options representing “Yes” and “No” between rounds (see Table 5).

Across rounds, participants rated the helpfulness of the weekly feedback messages, on average, 8.5 (SD 2.7) out of 10. Round 1 participants had two concerns with the weekly adherence feedback text message. First, many participants felt these messages were wordy and confusing. Because the message provided numerical information about the number of days a participant took his or her medication in both the past and the prior week, the content was difficult to read and interpret. We simplified feedback messages by including only the number of adherent days from the past week, but we indicated whether adherence had improved, stayed the same, or declined with an encouraging statement (Table 5).

Second, some round 1 participants complained their feedback underreported their adherence. This was due, in part, to the limited number of accepted responses to the assessment message (described above). However, we also learned the system was not counting responses received after midnight on the day it sent this message, so we extended the response time window (Table 5). In round 1, participants rated the helpfulness of feedback messages 8.2 (SD 3.0) out of 10. After revising feedback text messages and resolving functionality issues, participants in subsequent rounds rated the helpfulness of these messages 8.8 (SD 2.2) out of 10.

Across rounds, very few participants accessed their HbA_1c test result. One participant logged into the HbA_1c website and 2 participants called the REACH Helpline to get the result over the phone. Participants’ most common reason for not using either option was that they learned their HbA_1c test result from their clinic before receiving the HbA_1c text message. When asked their opinion about accessing their result with the HbA_1c text message, some participants appreciated this convenience, whereas others preferred their health care provider contact them with the result. On the basis of this feedback and the feedback from providers who preferred delivering and individually interpreting HbA_1c test results, we reduced our interpretation of the HbA_1c test result on the website (Table 5) and over the phone.

We received 22 voice mails on the REACH Helpline (12 research-related, 8 technical-related, and 2 medication-related voice mails) from 11 participants (8 of whom called more than once). When we asked the other 25 participants about the Helpline, most said they simply did not need it but thought they might use it if the program lasted longer.

On the basis of participant feedback and lessons learned by research staff, we made several changes to our research process. One change involved modifying how we administered the barrier assessment. Initially, research assistants asked participants to rate how much each of the barrier items gets in the way of taking their medication by reading each item aloud sequentially and asking for a rating. After round 1, research assistants reported some participants became disinterested/disengaged when completing this assessment and 20% reported no barriers despite having suboptimal HbA_1c levels. Therefore, after round 1, we changed the barrier assessment to a two-stage process. The first stage is a card-sorting task in which participants sort cards with barrier statements printed on them (see Measures section) into piles labeled “Sometimes” or “Never” based on whether or not the barrier applies to them. Next, research assistants ask participants to rate the degree to which each barrier placed in the “Sometimes” pile applies to them from 1=“a little” to 10=“a lot.” Before the two-stage process, round 1 participants reported a total of 62 barriers. After implementing the two-stage process, round 2 and round 3 participants reported 87 and 92 barriers, respectively. According to research assistants, participants in rounds 2 and 3 were more engaged during the barrier assessment process than participants in round 1.

We also modified the instructions provided during enrollment. Round 1 participants did not know they could change the timing of their text messages, so, in subsequent rounds, we clarified that participants could call the REACH Helpline at any point to request a time change. Also, during round 1, many participants sent unprompted responses (eg, “Thanks” or “OK, I will”) to the text messages promoting self-care, suggesting they thought a person sent these messages. Therefore, we revised our informed consent to make clear that a computer was sending text messages and not a person. For additional safeguarding, MEMOTEXT monitors all text message responses and notifies the REACH team if any text message requires follow-up.

Text messaging interventions provide an opportune platform for extending the delivery of tailored diabetes education and support; however, few have been designed for and tested among disadvantaged persons with T2DM [27,28]. We developed REACH—a tailored text messaging intervention designed to overcome user-specific medication adherence barriers and support other self-care behaviors—and tested its usability among patients with T2DM who were representative of the population REACH is designed for (ie, racially diverse, low SES, more than 25% limited health literacy). Participants who experienced REACH for 2 weeks had favorable opinions and responded frequently to daily text messages. We learned participants’ concerns/preferences, technical issues, and problems with our research process that we then fixed between each testing round, improving REACH in preparation for an evaluative trial.

Overall, participants were satisfied with REACH and provided favorable ratings for each of its elements. Text messages provided emotional/social support, reminded participants to engage in self-care activities, and helped them keep their self-care routine on track. In a similar 4-week study, Dick et al [71] assessed the usability of a text messaging program (SMS-DMCare) for improving T2DM self-care among African Americans. Participants provided ratings comparable to REACH regarding SMS-DMCare’s ease of use and provided similar interview feedback (eg, messages were helpful by reminding participants to take medication amid the demands of their daily lives) [71]. REACH’s text message engagement was higher than SMS-DMCare’s engagement [71], which may be due to REACH’s tailored content and/or personalized adherence feedback.

Usability studies often rely on survey- or questionnaire-based feedback [72], which may overlook much of what participants like or do not like about a system and how to improve it. Georgsson and Staggers [73] endorse using multiple data sources to identify and address usability issues. We collected both quantitative and qualitative feedback and system-collected data to fully understand users’ experience, improve our programmatic content and functionality, and resolve technical problems. REACH users who were adults from racially diverse and low-SES groups expressed concerns with the phrasing and wordiness of some messages, so we improved them. Furthermore, unanticipated feedback from research assistants and clinic staff was instrumental in refining our research process.

Through this multiple data source approach to usability testing, we made several improvements to REACH and enriched the user experience. We revised text messages to be more comprehensive, clear, and consistent with participants’ preferences. We limited our interpretation of HbA_1c test results to be more respectful of provider-patient relationships. We also improved REACH’s functionality to ensure the system recognizes and records participants’ attempts to interact with the intervention. Finally, we improved our assessment for capturing participants’ adherence barriers and modified our informed consent to ensure participants know how to use each intervention component and that a computer, not a person, sends all text messages.

There are several limitations to this study. First, participants experienced REACH for 2 weeks. Therefore, feedback and engagement may not be representative of participants experiencing REACH for longer periods. Furthermore, we compensated participants US $54 for their participation in the 2-week usability testing and feedback interview. By providing this incentive, we sought to adequately compensate participants for time and travel to the enrollment appointment and to offset mobile phone costs associated with text messages and the phone interview. This compensation may have inflated engagement with the intervention, but it was important for usability testing that participants actually use the intervention to be able to provide meaningful feedback. Despite this compensation, few participants accessed the REACH Helpline and HbA_1c website, making it difficult to gain insight on how participants felt about these elements and their functionality. Furthermore, participants may have been reluctant to provide critical feedback owing to study compensation, social desirability, or associating the study with their clinic. Additionally, because we were interested in specific questions concerning each intervention element, we composed our own feedback interview items and do not have validity and reliability information to report. Finally, although our sample size far exceeded the targeted enrollment for qualitative (at least 5) and quantitative (at least 20) usability testing [74,75], our sample was still too small to examine differences in opinions by participant characteristics.

Usability testing is imperative for ensuring that effects identified during efficacy trials are due to the intervention as intended and not due to errors in understanding or using the intervention [72]. Moreover, involving disadvantaged adults in usability testing may reveal preferences and concerns unique to this population. Iterative usability testing of the REACH intervention using multiple data sources revealed shortcomings in content, functionality, and research processes that we addressed before evaluating its effects on adherence and glycemic control in a randomized controlled trial. The REACH randomized controlled trial will assess the intervention’s effectiveness by recruiting patients from FQHCs and comparing outcomes between patients who did and did not experience REACH. To gain knowledge about REACH’s potential for implementation in clinic settings as a supplement to usual care, we will compensate participants for completing study assessments but will not provide them mobile phones or mobile phone plans. Although users experienced and commented on REACH specifically, our usability testing process and findings are applicable to other technology-delivered health interventions for disadvantaged populations. Specifically, informed consent should affirm that participants understand an intervention’s functionality, capabilities, and automation. Additionally, intervention content should provide enough information to be useful and avoid implying that a user is experiencing a specific issue; this is challenging while maintaining brevity necessary for digital content. Finally, our findings emphasize the importance of using multiple methods and sources of data to identify and resolve usability issues.