JMIR Publications

JMIR Human Factors

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 08.09.16 in Vol 3, No 2 (2016): Jul-Dec

This paper is in the following e-collection/theme issue:

    Original Paper

    Development and Usability of REACH: A Tailored Theory-Based Text Messaging Intervention for Disadvantaged Adults With Type 2 Diabetes

    1Department of Medicine, Vanderbilt University Medical Center, Nashville, TN, United States

    2Center for Health Behavior and Health Education, Vanderbilt University Medical Center, Nashville, TN, United States

    3Center for Diabetes Translational Research, Vanderbilt University Medical Center, Nashville, TN, United States

    4School of Nursing, Vanderbilt University, Nashville, TN, United States

    5Center for Clinical Quality and Implementation Research, Vanderbilt University Medical Center, Nashville, TN, United States

    6Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States

    The REACH Team

    Corresponding Author:

    Chandra Y Osborn, MPH, PhD

    Department of Medicine

    Vanderbilt University Medical Center

    2525 West End Ave

    Suite 370

    Nashville, TN, 37203

    United States

    Phone: 1 6159368468

    Fax:1 6158752201

    Email:


    ABSTRACT

    Background: Among adults with type 2 diabetes mellitus (T2DM), adherence to recommended self-care activities is suboptimal, especially among racial and ethnic minorities with low income. Self-care nonadherence is associated with having worse glycemic control and diabetes complications. Text messaging interventions are improving the self-care of adults with T2DM, but few have been tested with disadvantaged populations.

    Objective: To develop Rapid Education/Encouragement And Communications for Health (REACH), a tailored, text messaging intervention to support the self-care adherence of disadvantaged patients with T2DM, based on the Information-Motivation-Behavioral skills model. We then tested REACH’s usability to make improvements before evaluating its effects.

    Methods: We developed REACH’s content and functionality using an empirical and theory-based approach, findings from a previously pilot-tested intervention, and the expertise of our interdisciplinary research team. We recruited 36 adults with T2DM from Federally Qualified Health Centers to participate in 1 of 3 rounds of usability testing. For 2 weeks, participants received daily text messages assessing and promoting self-care, including tailored messages addressing users’ unique barriers to adherence, and weekly text messages with adherence feedback. We analyzed quantitative and qualitative user feedback and system-collected data to improve REACH.

    Results: Participants were, on average, 52.4 (SD 9.5) years old, 56% (20/36) female, 63% (22/35) were a racial or ethnic minority, and 67% (22/33) had an income less than US $35,000. About half were taking insulin, and average hemoglobin A1c level was 8.2% (SD 2.2%). We identified issues (eg, user concerns with message phrasing, technical restrictions with responding to assessment messages) and made improvements between testing rounds. Overall, participants favorably rated the ease of understanding (mean 9.6, SD 0.7) and helpfulness (mean 9.3, SD 1.4) of self-care promoting text messages on a scale of 1-10, responded to 96% of assessment text messages, and rated the helpfulness of feedback text messages 8.5 (SD 2.7) on a scale of 1-10. User feedback led to refining our study enrollment process so that users understood the flexibility in message timing and that computers, not people, send the messages. Furthermore, research assistants’ feedback on the enrollment process helped improve participants’ engagement with study procedures.

    Conclusions: Testing technology-delivered interventions with disadvantaged adults revealed preferences and concerns unique to this population. Through iterative testing and multiple data sources, we identified and responded to users’ intervention preferences, technical issues, and shortcomings in our research procedures.

    JMIR Hum Factors 2016;3(2):e23

    doi:10.2196/humanfactors.6029

    KEYWORDS



    Introduction

    Overview

    Currently, at least one in three people will develop type 2 diabetes mellitus (T2DM) in his or her lifetime [1]. People with diabetes are at higher risk of critical health complications including kidney failure, heart disease, and stroke [1]. More than 20% of health care spending in the United States goes toward people with a diagnosis of diabetes [1]. Racial and ethnic minorities are more likely than non-Hispanic whites to have a diagnosis of T2DM [1] and, once diagnosed, have more diabetes-related complications [2], hospitalizations [3], and premature death [4].

    People with T2DM can take medication, eat healthily, exercise, and test blood glucose levels to achieve optimal glycemic control [5] and, in turn, prevent diabetes complications [6] and premature mortality [7]. However, the initiation and maintenance of these self-care activities is challenging [8], and rates of self-care adherence are low among adults with T2DM [8,9]. Adherence rates are even lower among racial and ethnic minorities [10-12] and persons of low socioeconomic status (SES) [13] owing in part to financial difficulties and misconceptions about diabetes and self-care (eg, believing they do not have control over their diabetes, believing medication is not important) [8,14-16].

    Mobile phone–based interventions using text messaging are a practical approach for improving medication adherence among low-SES, racial and ethnic minorities with T2DM. More than 90% of US adults own a mobile phone [17]. Although smartphones are used less among individuals with diabetes [18], lower income [19], and lower education [19], text messaging does not require a smartphone and is the most common activity among all mobile phone users, used equally across SES, race, and ethnicity strata [20,21].

    Text messaging interventions are improving the self-care and glycemic control of adults with diabetes [22-24], but few have been tested with disadvantaged populations in the United States [25]. Two prior text messaging interventions [26,27] improved glycemic control in low-SES samples but not relative to a control group. A third text messaging intervention [28] improved glycemic control among a racially diverse sample but this sample had relatively high SES. Each of these interventions use text messages to address barriers to self-care, but none identifies and addresses each user’s unique barriers. Such barriers vary from person to person with T2DM [29-32], requiring a tailored user experience.

    Tailoring text messages to a user’s unique adherence barriers can address issues most applicable to him or her, such as limited diabetes knowledge, negative beliefs about medication (eg, fear of side effects), or limited financial resources [16]. Although interventions cannot easily target a person’s SES, they can enhance one’s problem-solving ability to address financial barriers and other modifiable barriers [14,16]. We developed the MEssaging for Diabetes (MED) intervention that sends tailored text messages addressing user-specific barriers to adherence and text messages assessing adherence [33]. After 3 months of MED among disadvantaged adults with T2DM, users’ barriers were reduced and barrier reduction was associated with improved glycemic control [34]. Furthermore, MED users were highly engaged, responding to 84% of daily assessment messages, and engagement did not differ by sex, race, income, health literacy, or duration of diagnosed diabetes [35]. MED’s findings are consistent with reviews suggesting text messaging interventions with personally relevant, tailored content are more engaging [36] and effective [37] than those without tailored content.

    Additionally, there is mixed evidence as to whether theory-based interventions are more effective than atheoretical interventions [38,39]; this is in part due to interventions not extensively applying theory and using theories that are inappropriate for behavior change [38]. The Information-Motivation-Behavioral skills (IMB) model suggests that adherence to a behavior depends on behavior-specific knowledge, personal and social motivation, and behavioral skills [40]. The IMB model is empirically validated among a wide range of diverse samples of adults with T2DM, including samples with low SES [41-43], and explains more than 40% of the variance in their medication adherence [43]. Text messaging interventions for T2DM self-care are rarely based on health behavior theory [24,44,45]. Of the few text messaging interventions tested among disadvantaged populations with T2DM, only 2 mention using a theory-driven approach [27,28]; however, the extent to which theory was applied in these interventions is unclear.

    Objective

    We developed Rapid Education/Encouragement And Communications for Health (REACH), a tailored, IMB model–based text messaging intervention. We performed 3 rounds of usability testing with adults with T2DM receiving care from Federally Qualified Health Centers (FQHCs) to identify and address any content and functionality issues before evaluating REACH’s effects on self-care and glycemic control.


    Methods

    REACH Intervention Development

    REACH was developed with MEMOTEXT, an algorithmic communications and data management platform supporting personalized user outputs and inputs via short message service (SMS); interventions using this platform have been tested with diverse patient populations with different health conditions who found them acceptable, engaging, and whose adherence improved >30% [46,47]. We worked with MEMOTEXT to develop REACH based on our experience developing and testing MED [33-35].

    REACH Content Development

    Similar to MED, we created tailored text messages addressing barriers to medication adherence common in our target population; however, REACH addresses more barriers to adherence than MED and barriers map onto the IMB model. To develop REACH content, we first conducted a thorough review of published studies reporting medication adherence barriers among adults with T2DM. In October 2014, we searched for studies in PubMed using terms from each of 3 categories: (1) medication adherence (ie, diabetes medication, medication adherence, medication nonadherence, medication compliance), (2) barriers (ie, barriers, challenges, problems), and (3) diabetes or type 2 diabetes. Terms were intralinked with “OR” and “AND.” There were no restrictions on year of publication. We then searched references cited in eligible articles and articles citing relevant articles by hand. Experts in diabetes medication adherence on our team (authors SK and CYO) ensured our search captured meaningful articles. We reviewed all studies identifying barriers to diabetes medication adherence among adults diagnosed with T2DM and documented the reported barriers and race and ethnicity of the sample. Across 30 studies, we identified 68 barriers to taking medications and 7 barriers to taking insulin. We then sorted and collapsed similar barriers, resulting in 31 medication-related and 5 insulin-specific barriers. Finally, we tagged each of the 36 barriers to the IMB model’s information, motivation, or behavioral skills constructs [40], and content experts drew on identified studies to develop text messages addressing each barrier (Table 1).

    Users of MED wanted text messages providing information specific to their prescribed medications. Therefore, the REACH team’s clinical pharmacist and nurse practitioner identified and classified available oral hypoglycemic agents, insulin, and noninsulin injectable drugs (glucagon-like peptide-1 receptor agonists). They then developed regimen-specific text messages on how to handle missed doses, manage medication side effects, administer medication, and store and discard medication for each class of medication.

    MED users also recommended adding messages promoting other self-care behaviors (in addition to medication adherence) and inspirational messages [67]. In response, the REACH team’s dietitian/diabetes educator developed text messages with tips promoting adherence to healthful eating, physical activity, and self-monitoring of blood glucose (SMBG). These messages were developed with the goal of providing general diabetes nutrition, exercise, and SMBG statements that are applicable to people with diabetes (vs specific instructions or information that should be determined in a one-on-one consult). Therefore, guidelines for development of these messages were to generate content providing concrete and practical diabetes information applicable to most adults with T2DM. We also developed inspirational text messages to encourage the initiation and maintenance of self-care efforts (eg, “Remember that you have the power every day to make progress toward improving your health!”) and ensured all messages were contextually appropriate (eg, referenced local resources, avoided mention of things such as gym memberships).

    After developing all content, the REACH team’s health communication experts reviewed and edited text messages to be readable and understandable (ie, written at the sixth-grade reading level, avoided complex terms and jargon, and health literacy appropriate). Finally, a digital content developer shortened messages (≤160 characters) and ensured consistent tone across messages and appropriateness for digital delivery.

    REACH Functionality Development

    With the help of MEMOTEXT, we developed functionality to optimize the REACH user experience, making it more personalized and interactive than MED. MED users received a daily text message assessing whether they took their medication that day. Users responded to this message frequently [35] and said it served as a reminder to take their medication [67]. We retained this feature in REACH but made the experience more interactive. For example, if users respond “no,” they receive a follow-up message asking why they did not take their medication with response options to encourage reflection on reasons for nonadherence (Figure 1).

    MED users received feedback on their adherence (ie, aggregated responses to daily adherence assessment messages) via an interactive voice response (IVR) call. Although users enjoyed receiving adherence feedback, most said the IVR call was a nuisance [67]. They were also less likely to answer calls than respond to text messages [35]. Therefore, REACH delivers adherence feedback via a weekly text message instead of a weekly IVR call. Feedback reflects participants’ adherence for the past week and for the prior week and delivers an encouraging message tailored to whether adherence improved, declined, or stayed the same.

    Finally, MED users wanted to change the times they received text messages, so REACH allows for flexibility in text message timing. Users determine a preferred window of time to receive self-care promoting text messages and indicate their bedtime for receiving adherence assessment text messages. Participants are able to adjust message timing throughout the intervention by contacting the REACH Helpline (described below).

    Table 1. Information, Motivation, and Behavioral skills (IMB) barriers to medication adherence for patients with type 2 diabetes mellitus identified through a literature review.
    View this table

    MEMOTEXT tailors, schedules, and sends text messages using participant data received through an application programming interface (API). At enrollment, research assistants enter participants’ survey responses and electronic health record (EHR) data into REDCap, a secure, Web-based application designed to support data capture for multisite studies [68,69]. REDCap data are then transferred to MEMOTEXT via the API. MEMOTEXT tailors the messages addressing medication adherence barriers by ranking participants’ self-reported barrier scores (see Measures section) and sending messages addressing each user’s 4 highest-ranked barriers. In instances of a tie, the system randomly selects among the tied barriers. MEMOTEXT also tailors regimen-specific messages based on each user’s prescribed diabetes medication taken from the EHR.

    We describe each REACH component in Table 2. REACH users receive 2 daily text messages: (1) a text promoting self-care—either tailored to user-identified barriers to medication adherence or nontailored to promote another self-care behavior—and (2) a text assessing medication adherence for that day. Users also receive a weekly text message with medication adherence feedback based on responses to daily assessment texts. Furthermore, after users have their hemoglobin A1c (HbA1c) level tested during study enrollment, they receive a text message providing directions on how to access their HbA1c test result; users can either log on to a Health Insurance Portability and Accountability Act (HIPAA)–compliant webpage hosted by MEMOTEXT or, if they do not have access to Internet, call the REACH Helpline. Finally, users have access to the REACH Helpline for research-, technical-, and medication-related questions. When users leave a voicemail on the Helpline, a REACH team member returns their call within one business day. Figure 2 illustrates the REACH user experience with example text messages.

    Table 2. Rapid Education/Encouragement And Communications for Health intervention components (REACH).
    View this table
    Figure 1. Functionality for adherence assessment text message.
    View this figure
    Figure 2. Rapid Education/Encouragement And Communications for Health (REACH) experience for a hypothetical user. Each medication adherence (blue circle) text message (3 per week) addresses one of the user’s top 4 barriers to medication adherence. IMB: information-Motivation-Behavioral skills; SMBG: self-monitoring of blood glucose; A1c: hemoglobin A1c.
    View this figure

    Usability Testing

    Sample and Recruitment

    Using flyers, interest cards, and referrals from clinic staff, we recruited participants from FQHCs in Nashville, Tennessee. Eligible participants had a T2DM diagnosis, were currently prescribed at least one daily diabetes medication, were responsible for taking their diabetes medication (ie, a caregiver did not administer medication), had a mobile phone with text messaging, were at least 18 years of age, could speak and read English, and provided a social security number (necessary to process compensation). Exclusion criteria included an existing diagnosis of dementia, auditory limitations, an inability to communicate orally, and an inability to receive, read, or send a text message as determined by trained research assistants.

    Data and Procedures

    The Vanderbilt University Institutional Review Board approved all study procedures before enrollment. Research assistants met with interested patients to describe the study and verify eligibility. In a private room at the patient’s clinic, research assistants administered a brief cognitive screening instrument [70] and sent a test text message to each patient to assess whether he or she could see, read, and successfully respond to the message. If a patient passed this screener, research assistants obtained informed consent before verbally administering survey measures. A clinic phlebotomist performed a blood-drawn HbA1c test. Research assistants accessed participants’ EHRs to confirm a T2DM diagnosis, collect the type and quantity of prescribed diabetes medication, and the study HbA1c test result.

    During each testing round, participants experienced REACH for 2 weeks and then completed a semistructured phone interview that qualitatively assessed their user experience. Following each round, research assistants collected user feedback, the REACH team resolved content- and research-related issues, and MEMOTEXT resolved technical issues before the next round. Participants received up to US $54 for completing the enrollment survey (US $20), replying to assessment messages (US $1/day), and completing the phone interview (US $20).

    Measures
    Sample Characteristics

    We collected self-reported age, sex, race, ethnicity, income, education (ie, years in school), insulin status, and diabetes duration (ie, years since a diabetes diagnosis). We also asked about comfort with mobile phones and text messaging and used validated survey instruments to capture additional information.

    Barriers to Medication Adherence

    Respondents rated how much each barrier in Table 1 (written as statements, eg, “I’m not sure what my diabetes medicine is supposed to do”) gets in the way of taking their diabetes medication from 1=“not at all” to 10=“a lot.” Each item maps onto a single IMB model–based barrier.

    REACH Engagement

    We measured engagement with system-collected responses to daily adherence assessment texts, the frequency of REACH Helpline calls, and the frequency of accessing HbA1c test results via the website. We calculated engagement with assessment text messages by dividing each participant’s number of responses by the total number of messages sent to him or her. MEMOTEXT tracked Helpline calls and HbA1c website use.

    User Feedback

    Likert-type items assessed ease of understanding and helpfulness of the REACH intervention elements. Open-ended items assessed what users did and/or did not like, asked how and why an element was or was not helpful, and elicited suggestions for improving the REACH user experience. Table 3 presents items used to elicit user feedback.

    Analyses

    We calculated descriptive statistics with SPSS Statistics version 23 (IBM Corp). Interviews were audio-recorded and transcribed verbatim by an external transcription service. Questions and responses were pasted into REDCap under each interview question, organized by testing round, and then exported to Excel. We undertook a pragmatic approach to analyze participant feedback quickly between rounds to support changes to the intervention in a timely fashion. Between rounds, a member of the research team (LAN) read interview transcripts to manually categorize participants’ feedback by intervention component. We then looked across participants’ comments for each intervention component to identify areas for improvement and to ascertain the overall tone and message of the users’ feedback for each component.

    Table 3. User feedback interview items by intervention element.
    View this table

    Results

    Participant characteristics

    An average of 12 participants experienced REACH each testing round, totaling 36 participants (Table 4). The average age of the participants was 52.4 (SD 9.5) years, 63% (22/35) were a racial or ethnic minority, 39% (14/36) had less than a high school degree or equivalent, and 67% (22/33) had an income less than US $35,000. The average HbA1c level was 8.2% (SD 2.2%); 64% (23/36) of the participants had suboptimal glycemic control (HbA1c≥7.0%). Across rounds, the most frequently reported barriers to medication adherence were forgetting to take doses (56%, 20/36 users reported this barrier with an average score of 5.2, SD 3.0, of 10), the high cost of medication (44%, 16/36 users; mean score 6.2, SD 2.7, of 10), and believing that taking medication is unpleasant (42%, 15/36 users; mean score 5.2, SD 2.6, of 10). The most commonly reported insulin-specific barrier was feeling embarrassed to take insulin in public (35%, 6/17 users who were prescribed insulin; mean score 4.5, SD 2.4, of 10).

    Table 4. Participant characteristics.
    View this table

    All 36 participants completed an exit interview. Overall, participants said REACH was helpful and gave favorable feedback on each intervention element. Participants reported preferences and technical issues requiring iterative improvements between testing rounds. Below, we describe this feedback, our iterative changes by intervention element, followed by changes in our research processes.

    Daily Text Message Promoting Self-Care

    Across rounds, participants rated the ease of understanding and helpfulness of the daily self-care text message, on average, 9.6 (SD 0.7) and 9.3 (SD 1.4) on a scale of 1-10, respectively. Participants appreciated that these messages were simple and without medical jargon. Participants said inspirational messages made them feel supported and not alone in living with diabetes and motivated them to take more initiative with self-care. Messages with self-care tips and information were helpful because they either provided a useful reminder or communicated something new.

    Interviewer: Why did you read those messages? What made you want to?

    Participant: They was [sic] helpful. Some things I didn't know [sic]. They helped me understand a lot of stuff because I didn't understand. [37-year-old, African American male]

    Many participants valued reminders to care for their diabetes when they otherwise might not think about it:

    I thought these messages were very helpful. I get so busy in the day that I don’t [even] take time to eat. And then when I get a text, [I realize] oh, wow, I need to do something. That really helps a lot. I wish I had somebody who did that for me all the time. [59-year-old, white male]

    Despite the overall positive feedback about self-care messages, some participants had concerns. For example, a round 2 participant said a message provided a suggestion for remembering to take medications, without providing the steps for carrying out the suggestion. A few participants said some messages implied a problem when they did not have one (eg, “Struggling to take your diabetes medications every day? Talk to a loved one about what is getting in your way.”). To address such concerns, we revised all problematic text messages between rounds 2 and 3 to provide additional context and be less presumptuous (see Table 5 for examples of problematic and revised text messages).

    Daily Text Message Assessing Adherence

    Across rounds, participants responded to 96% of adherence assessment text messages and, on average, rated the helpfulness of these messages 9.1 (SD 2.1) out of 10. Assessment messages helped remind participants to take their medications and maintain their routine. One participant commented on these messages’ emotional and social support:

    [The texts] keep you on task about what you should do...especially if someone doesn't have anybody around. You know it's kind of like having a family member around to remind you, “Hey, you should take your meds.” These [texts] make you feel like someone cares or is concerned about your health and makes sure you're taking care of yourself. So I think that's very helpful. [48-year-old, African American male]

    Several participants with optimal glycemic control (HbA1c <7%) said these messages were not particularly helpful because they routinely took their medication and rarely missed doses. Nonetheless, these participants endorsed the value of these messages for others recently diagnosed with diabetes and/or newly prescribed medication who do not have an established routine.

    Round 1 participants complained about needing to respond to assessment messages several times before the system would accept their response. Upon viewing system-collected data, we learned that participants used different variations of “Yes” to respond (eg, “Yup” or “Yeah”). Therefore, we expanded the acceptable response options representing “Yes” and “No” between rounds (see Table 5).

    Weekly Text Message Providing Feedback

    Across rounds, participants rated the helpfulness of the weekly feedback messages, on average, 8.5 (SD 2.7) out of 10. Round 1 participants had two concerns with the weekly adherence feedback text message. First, many participants felt these messages were wordy and confusing. Because the message provided numerical information about the number of days a participant took his or her medication in both the past and the prior week, the content was difficult to read and interpret. We simplified feedback messages by including only the number of adherent days from the past week, but we indicated whether adherence had improved, stayed the same, or declined with an encouraging statement (Table 5).

    Second, some round 1 participants complained their feedback underreported their adherence. This was due, in part, to the limited number of accepted responses to the assessment message (described above). However, we also learned the system was not counting responses received after midnight on the day it sent this message, so we extended the response time window (Table 5). In round 1, participants rated the helpfulness of feedback messages 8.2 (SD 3.0) out of 10. After revising feedback text messages and resolving functionality issues, participants in subsequent rounds rated the helpfulness of these messages 8.8 (SD 2.2) out of 10.

    Hemoglobin A1c Text Message

    Across rounds, very few participants accessed their HbA1c test result. One participant logged into the HbA1c website and 2 participants called the REACH Helpline to get the result over the phone. Participants’ most common reason for not using either option was that they learned their HbA1c test result from their clinic before receiving the HbA1c text message. When asked their opinion about accessing their result with the HbA1c text message, some participants appreciated this convenience, whereas others preferred their health care provider contact them with the result. On the basis of this feedback and the feedback from providers who preferred delivering and individually interpreting HbA1c test results, we reduced our interpretation of the HbA1c test result on the website (Table 5) and over the phone.

    Table 5. Changes made to the Rapid Education/Encouragement And Communications for Health intervention during usability testing (REACH).
    View this table

    REACH Helpline

    We received 22 voice mails on the REACH Helpline (12 research-related, 8 technical-related, and 2 medication-related voice mails) from 11 participants (8 of whom called more than once). When we asked the other 25 participants about the Helpline, most said they simply did not need it but thought they might use it if the program lasted longer.

    Research Processes

    On the basis of participant feedback and lessons learned by research staff, we made several changes to our research process. One change involved modifying how we administered the barrier assessment. Initially, research assistants asked participants to rate how much each of the barrier items gets in the way of taking their medication by reading each item aloud sequentially and asking for a rating. After round 1, research assistants reported some participants became disinterested/disengaged when completing this assessment and 20% reported no barriers despite having suboptimal HbA1c levels. Therefore, after round 1, we changed the barrier assessment to a two-stage process. The first stage is a card-sorting task in which participants sort cards with barrier statements printed on them (see Measures section) into piles labeled “Sometimes” or “Never” based on whether or not the barrier applies to them. Next, research assistants ask participants to rate the degree to which each barrier placed in the “Sometimes” pile applies to them from 1=“a little” to 10=“a lot.” Before the two-stage process, round 1 participants reported a total of 62 barriers. After implementing the two-stage process, round 2 and round 3 participants reported 87 and 92 barriers, respectively. According to research assistants, participants in rounds 2 and 3 were more engaged during the barrier assessment process than participants in round 1.

    We also modified the instructions provided during enrollment. Round 1 participants did not know they could change the timing of their text messages, so, in subsequent rounds, we clarified that participants could call the REACH Helpline at any point to request a time change. Also, during round 1, many participants sent unprompted responses (eg, “Thanks” or “OK, I will”) to the text messages promoting self-care, suggesting they thought a person sent these messages. Therefore, we revised our informed consent to make clear that a computer was sending text messages and not a person. For additional safeguarding, MEMOTEXT monitors all text message responses and notifies the REACH team if any text message requires follow-up.


    Discussion

    Principal Findings

    Text messaging interventions provide an opportune platform for extending the delivery of tailored diabetes education and support; however, few have been designed for and tested among disadvantaged persons with T2DM [27,28]. We developed REACH—a tailored text messaging intervention designed to overcome user-specific medication adherence barriers and support other self-care behaviors—and tested its usability among patients with T2DM who were representative of the population REACH is designed for (ie, racially diverse, low SES, more than 25% limited health literacy). Participants who experienced REACH for 2 weeks had favorable opinions and responded frequently to daily text messages. We learned participants’ concerns/preferences, technical issues, and problems with our research process that we then fixed between each testing round, improving REACH in preparation for an evaluative trial.

    Overall, participants were satisfied with REACH and provided favorable ratings for each of its elements. Text messages provided emotional/social support, reminded participants to engage in self-care activities, and helped them keep their self-care routine on track. In a similar 4-week study, Dick et al [71] assessed the usability of a text messaging program (SMS-DMCare) for improving T2DM self-care among African Americans. Participants provided ratings comparable to REACH regarding SMS-DMCare’s ease of use and provided similar interview feedback (eg, messages were helpful by reminding participants to take medication amid the demands of their daily lives) [71]. REACH’s text message engagement was higher than SMS-DMCare’s engagement [71], which may be due to REACH’s tailored content and/or personalized adherence feedback.

    Usability studies often rely on survey- or questionnaire-based feedback [72], which may overlook much of what participants like or do not like about a system and how to improve it. Georgsson and Staggers [73] endorse using multiple data sources to identify and address usability issues. We collected both quantitative and qualitative feedback and system-collected data to fully understand users’ experience, improve our programmatic content and functionality, and resolve technical problems. REACH users who were adults from racially diverse and low-SES groups expressed concerns with the phrasing and wordiness of some messages, so we improved them. Furthermore, unanticipated feedback from research assistants and clinic staff was instrumental in refining our research process.

    Through this multiple data source approach to usability testing, we made several improvements to REACH and enriched the user experience. We revised text messages to be more comprehensive, clear, and consistent with participants’ preferences. We limited our interpretation of HbA1c test results to be more respectful of provider-patient relationships. We also improved REACH’s functionality to ensure the system recognizes and records participants’ attempts to interact with the intervention. Finally, we improved our assessment for capturing participants’ adherence barriers and modified our informed consent to ensure participants know how to use each intervention component and that a computer, not a person, sends all text messages.

    There are several limitations to this study. First, participants experienced REACH for 2 weeks. Therefore, feedback and engagement may not be representative of participants experiencing REACH for longer periods. Furthermore, we compensated participants US $54 for their participation in the 2-week usability testing and feedback interview. By providing this incentive, we sought to adequately compensate participants for time and travel to the enrollment appointment and to offset mobile phone costs associated with text messages and the phone interview. This compensation may have inflated engagement with the intervention, but it was important for usability testing that participants actually use the intervention to be able to provide meaningful feedback. Despite this compensation, few participants accessed the REACH Helpline and HbA1c website, making it difficult to gain insight on how participants felt about these elements and their functionality. Furthermore, participants may have been reluctant to provide critical feedback owing to study compensation, social desirability, or associating the study with their clinic. Additionally, because we were interested in specific questions concerning each intervention element, we composed our own feedback interview items and do not have validity and reliability information to report. Finally, although our sample size far exceeded the targeted enrollment for qualitative (at least 5) and quantitative (at least 20) usability testing [74,75], our sample was still too small to examine differences in opinions by participant characteristics.

    Conclusions

    Usability testing is imperative for ensuring that effects identified during efficacy trials are due to the intervention as intended and not due to errors in understanding or using the intervention [72]. Moreover, involving disadvantaged adults in usability testing may reveal preferences and concerns unique to this population. Iterative usability testing of the REACH intervention using multiple data sources revealed shortcomings in content, functionality, and research processes that we addressed before evaluating its effects on adherence and glycemic control in a randomized controlled trial. The REACH randomized controlled trial will assess the intervention’s effectiveness by recruiting patients from FQHCs and comparing outcomes between patients who did and did not experience REACH. To gain knowledge about REACH’s potential for implementation in clinic settings as a supplement to usual care, we will compensate participants for completing study assessments but will not provide them mobile phones or mobile phone plans. Although users experienced and commented on REACH specifically, our usability testing process and findings are applicable to other technology-delivered health interventions for disadvantaged populations. Specifically, informed consent should affirm that participants understand an intervention’s functionality, capabilities, and automation. Additionally, intervention content should provide enough information to be useful and avoid implying that a user is experiencing a specific issue; this is challenging while maintaining brevity necessary for digital content. Finally, our findings emphasize the importance of using multiple methods and sources of data to identify and resolve usability issues.

    Acknowledgments

    This research is funded by the National Institutes of Health NIH/NIDDK R01-DK100694. LSM is also supported by a career development award NIH/NIDDK K01-DK106306. CYO is also supported by a career development award NIH/NIDDK K01-DK087894.

    The authors thank Dr Chad K Gentry, Anne Brown, Becky P Gregory, Will Acuff, Kryseanna J Harper, David R Pagnotti Jr, Lauren M LeStourgeon, Sarah E Williamson, Tebeb Gebretsadik, Dr Tom A Elasy, and Dr Kevin B Johnson for their contributions to this research. We also thank personnel and participants at The Clinic at Mercury Courts, University Community Health Services, and Faith Family Medical Center.

    Authors' Contributions

    LAN codesigned the study, analyzed and coded data, and wrote the manuscript. LSM codesigned the study, oversaw data collection and analyses, and cowrote and edited the manuscript. KW codesigned the study and edited the manuscript. SK codesigned the study and edited the manuscript. EMB oversaw data collection and edited the manuscript. CYO codesigned the study, oversaw data collection and analyses, and cowrote and edited the manuscript.

    Conflicts of Interest

    SK is a consultant to SAI Interactive and Bioscape Digital. KW is an advisor to EdLogics, Inc.

    References

    1. Centers for Disease Control and Prevention. 2016. Diabetes: Working to reverse the US epidemic   URL: https://www.cdc.gov/chronicdisease/resources/publications/aag/pdf/2016/diabetes-aag.pdf [accessed 2016-08-01] [WebCite Cache]
    2. Osborn CY, de Groot M, Wagner JA. Racial and ethnic disparities in diabetes complications in the northeastern United States: the role of socioeconomic status. J Natl Med Assoc 2013;105(1):51-58 [FREE Full text] [Medline]
    3. Fayfman M, Vellanki P, Alexopoulos A, Buehler L, Zhao L, Smiley D, et al. Report on racial disparities in hospitalized patients with hyperglycemia and diabetes. The Journal of Clinical Endocrinology & Metabolism 2016;101(3):1144-1150. [CrossRef]
    4. Hunt BR, Whitman S, Henry CA. Age-adjusted diabetes mortality rates vary in local communities in a metropolitan area: racial and spatial disparities and correlates. Diabetes Care 2014;37(5):1279-1286. [CrossRef] [Medline]
    5. Centers for Disease Control and Prevention. 2014. Estimates of Diabetes and Its Burden in the United States   URL: https://www.cdc.gov/diabetes/pubs/statsreport14/national-diabetes-report-web.pdf [accessed 2016-08-31] [WebCite Cache]
    6. Asche C, LaFleur J, Conner C. A review of diabetes treatment adherence and the association with clinical and economic outcomes. Clin Ther 2011 Jan;33(1):74-109. [CrossRef] [Medline]
    7. Currie CJ, Peyrot M, Morgan CL, Poole CD, Jenkins-Jones S, Rubin RR, et al. The impact of treatment noncompliance on mortality in people with type 2 diabetes. Diabetes Care 2012 Jun;35(6):1279-1284 [FREE Full text] [CrossRef] [Medline]
    8. García-Pérez L, Alvarez M, Dilla T, Gil-Guillén V, Orozco-Beltrán D. Adherence to therapies in patients with type 2 diabetes. Diabetes Ther 2013 Dec;4(2):175-194 [FREE Full text] [CrossRef] [Medline]
    9. Kirkman MS, Rowan-Martin MT, Levin R, Fonseca VA, Schmittdiel JA, Herman WH, et al. Determinants of adherence to diabetes medications: findings from a large pharmacy claims database. Diabetes Care 2015 Apr;38(4):604-609 [FREE Full text] [CrossRef] [Medline]
    10. Egede LE, Gebregziabher M, Hunt KJ, Axon RN, Echols C, Gilbert GE, et al. Regional, geographic, and racial/ethnic variation in glycemic control in a national sample of veterans with diabetes. Diabetes Care 2011 Apr;34(4):938-943 [FREE Full text] [CrossRef] [Medline]
    11. Gebregziabher M, Lynch CP, Mueller M, Gilbert GE, Echols C, Zhao Y, et al. Using quantile regression to investigate racial disparities in medication non-adherence. BMC Med Res Methodol 2011;11:88 [FREE Full text] [CrossRef] [Medline]
    12. Capoccia K, Odegard PS, Letassy N. Medication adherence with diabetes medication: a systematic review of the literature. Diabetes Educ 2016 Feb;42(1):34-71. [CrossRef] [Medline]
    13. Rolnick SJ, Pawloski PA, Hedblom BD, Asche SE, Bruzek RJ. Patient characteristics associated with medication adherence. Clin Med Res 2013 Jun;11(2):54-65 [FREE Full text] [CrossRef] [Medline]
    14. Mann DM, Ponieman D, Leventhal H, Halm EA. Predictors of adherence to diabetes medications: the role of disease and medication beliefs. J Behav Med 2009 Jun;32(3):278-284. [CrossRef] [Medline]
    15. Mann DM, Ponieman D, Leventhal H, Halm EA. Misconceptions about diabetes and its management among low-income minorities with diabetes. Diabetes Care 2009 Apr;32(4):591-593 [FREE Full text] [CrossRef] [Medline]
    16. Nam S, Chesla C, Stotts NA, Kroon L, Janson SL. Barriers to diabetes management: patient and provider factors. Diabetes Res Clin Pract 2011 Jul;93(1):1-9. [CrossRef] [Medline]
    17. Raine L, Zickuhr K. Pew Research Center. 2015. Americans' views on mobile etiquette   URL: http://www.pewinternet.org/2015/08/26/americans-views-on-mobile-etiquette/ [accessed 2016-01-20] [WebCite Cache]
    18. Blondon KS, Hebert PL, Ralston JD. An exploration of the potential reach of smartphones in diabetes. AMIA Annu Symp Proc 2014;2014:289-296 [FREE Full text] [Medline]
    19. Smith A. Pew Research Center. 2015. U.S. Smartphone Use in 2015   URL: http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/ [accessed 2016-01-20] [WebCite Cache]
    20. Duggan M. Pew Research Center. 2013. Cell phone activities   URL: http:/​/www.​pewinternet.org/​files/​old-media/​Files/​Reports/​2013/​PIP_Cell%20Phone%20Activities%20May%202013.​pdf [accessed 2016-01-20] [WebCite Cache]
    21. Chakkalakal RJ, Kripalani S, Schlundt DG, Elasy TA, Osborn CY. Disparities in using technology to access health information: race versus health literacy. Diabetes Care 2014;37(3):e53-e54 [FREE Full text] [CrossRef] [Medline]
    22. Vervloet M, van Dijk L, Santen-Reestman J, van Vlijmen B, van Wingerden P, Bouvy ML, et al. SMS reminders improve adherence to oral medication in type 2 diabetes patients who are real time electronically monitored. Int J Med Inform 2012 Sep;81(9):594-604. [CrossRef] [Medline]
    23. Saffari M, Ghanizadeh G, Koenig HG. Health education via mobile text messaging for glycemic control in adults with type 2 diabetes: a systematic review and meta-analysis. Prim Care Diabetes 2014 Dec;8(4):275-285. [CrossRef] [Medline]
    24. Arambepola C, Ricci-Cabello I, Manikavasagam P, Roberts N, French DP, Farmer A. The impact of automated brief messages promoting lifestyle changes delivered via mobile devices to people with type 2 diabetes: a systematic literature review and meta-analysis of controlled trials. J Med Internet Res 2016;18(4):e86 [FREE Full text] [CrossRef] [Medline]
    25. Oldenburg B, Taylor CB, O'Neil A, Cocker F, Cameron LD. Using new technologies to improve the prevention and management of chronic conditions in populations. Annu Rev Public Health 2015 Mar 18;36:483-505. [CrossRef] [Medline]
    26. Capozza K, Woolsey S, Georgsson M, Black J, Bello N, Lence C, et al. Going mobile with diabetes support: a randomized study of a text message-based personalized behavioral intervention for type 2 diabetes self-care. Diabetes Spectr 2015 May;28(2):83-91. [CrossRef] [Medline]
    27. Arora S, Peters AL, Burner E, Lam CN, Menchine M. Trial to examine text message-based mHealth in emergency department patients with diabetes (TExT-MED): a randomized controlled trial. Ann Emerg Med 2014 Jun;63(6):745-54.e6. [CrossRef] [Medline]
    28. Nundy S, Dick JJ, Chou C, Nocon RS, Chin MH, Peek ME. Mobile phone diabetes project led to improved glycemic control and net savings for Chicago plan participants. Health Aff (Millwood) 2014 Feb;33(2):265-272 [FREE Full text] [CrossRef] [Medline]
    29. Bailey GR, Barner JC, Weems JK, Leckbee G, Solis R, Montemayor D, et al. Assessing barriers to medication adherence in underserved patients with diabetes in Texas. Diabetes Educ 2012;38(2):271-279. [CrossRef] [Medline]
    30. Tseng C, Tierney EF, Gerzoff RB, Dudley RA, Waitzfelder B, Ackermann RT, et al. Race/ethnicity and economic differences in cost-related medication underuse among insured adults with diabetes: the Translating Research Into Action for Diabetes Study. Diabetes Care 2008 Feb;31(2):261-266. [CrossRef] [Medline]
    31. Odegard PS, Gray SL. Barriers to medication adherence in poorly controlled diabetes mellitus. Diabetes Educ 2008;34(4):692-697. [CrossRef] [Medline]
    32. Osborn CY, Mayberry LS, Wagner JA, Welch GW. Stressors may compromise medication adherence among adults with diabetes and low socioeconomic status. West J Nurs Res 2014 Oct;36(9):1091-1110 [FREE Full text] [CrossRef] [Medline]
    33. Osborn CY, Mulvaney SA. Development and feasibility of a text messaging and interactive voice response intervention for low-income, diverse adults with type 2 diabetes mellitus. J Diabetes Sci Technol 2013 May;7(3):612-622 [FREE Full text] [Medline]
    34. Mayberry LS, Mulvaney SA, Johnson KB, Osborn CY. The MEssaging for Diabetes (MED) intervention improves short-term medication adherence among low-income adults with type 2 diabetes. Journal of Diabetes Science and Technology 2016:- Epub ahead of print(forthcoming). [Medline]
    35. Nelson LA, Mulvaney SA, Gebretsadik T, Ho Y, Johnson KB, Osborn CY. Disparities in the use of a mHealth medication adherence promotion intervention for low-income adults with type 2 diabetes. J Am Med Inform Assoc 2016 Jan;23(1):12-18. [CrossRef] [Medline]
    36. Nelson LA, Coston TD, Cherrington AL, Osborn CY. Patterns of user engagement with mobile- and web-delivered self-care interventions for adults with T2DM: a review of the literature. Curr Diab Rep 2016 Jul;16(7):66. [CrossRef] [Medline]
    37. Head KJ, Noar SM, Iannarino NT, Grant HN. Efficacy of text messaging-based interventions for health promotion: a meta-analysis. Soc Sci Med 2013 Nov;97:41-48. [CrossRef] [Medline]
    38. Prestwich A, Webb TL, Conner M. Using theory to develop and test interventions to promote changes in health behaviourvidence, issues, and recommendations. Current Opinion in Psychology 2015;5:1-5. [CrossRef]
    39. Glanz K, Bishop DB. The role of behavioral science theory in development and implementation of public health interventions. Annu Rev Public Health 2010;31:399-418. [CrossRef] [Medline]
    40. Fisher JD, Fisher WA, Amico KR, Harman JJ. An information-motivation-behavioral skills model of adherence to antiretroviral therapy. Health Psychol 2006 Jul;25(4):462-473. [CrossRef] [Medline]
    41. Osborn CY, Rivet AK, Fisher WA, Egede LE, Fisher JD. An information-motivation-behavioral skills analysis of diet and exercise behavior in Puerto Ricans with diabetes. J Health Psychol 2010 Nov;15(8):1201-1213 [FREE Full text] [CrossRef] [Medline]
    42. Osborn CY, Egede LE. Validation of an Information-Motivation-Behavioral Skills model of diabetes self-care (IMB-DSC). Patient Educ Couns 2010 Apr;79(1):49-54 [FREE Full text] [CrossRef] [Medline]
    43. Mayberry LS, Osborn CY. Empirical validation of the information-motivation-behavioral skills model of diabetes medication adherence: a framework for intervention. Diabetes Care 2014;37(5):1246-1253 [FREE Full text] [CrossRef] [Medline]
    44. Farmer AJ, McSharry J, Rowbotham S, McGowan L, Ricci-Cabello I, French DP. Effects of interventions promoting monitoring of medication use and brief messaging on medication adherence for people with Type 2 diabetes: a systematic review of randomized trials. Diabet Med 2015 Oct 15;33(5):565-579. [CrossRef] [Medline]
    45. DeKoekkoek T, Given B, Given CW, Ridenour K, Schueller M, Spoelstra SL. mHealth SMS text messaging interventions and to promote medication adherence: an integrative review. J Clin Nurs 2015 Oct;24(19-20):2722-2735. [CrossRef] [Medline]
    46. Boland MV, Chang DS, Frazier T, Plyler R, Jefferys JL, Friedman DS. Automated telecommunication-based reminders and adherence with once-daily glaucoma medication dosing: the automated dosing reminder study. JAMA Ophthalmol 2014 Jul;132(7):845-850. [CrossRef] [Medline]
    47. Boland MV, Chang DS, Frazier T, Plyler R, Friedman DS. Electronic monitoring to assess adherence with once-daily glaucoma medications and risk factors for nonadherence: the automated dosing reminder study. JAMA Ophthalmol 2014 Jul;132(7):838-844. [CrossRef] [Medline]
    48. Duru OK, Gerzoff RB, Selby JV, Brown AF, Ackermann RT, Karter AJ, et al. Identifying risk factors for racial disparities in diabetes outcomes: the translating research into action for diabetes study. Med Care 2009 Jun;47(6):700-706 [FREE Full text] [Medline]
    49. Nagelkerk J, Reick K, Meengs L. Perceived barriers and effective strategies to diabetes self-management. J Adv Nurs 2006 Apr;54(2):151-158. [CrossRef] [Medline]
    50. Hill-Briggs F, Gary TL, Bone LR, Hill MN, Levine DM, Brancati FL. Medication adherence and diabetes control in urban African Americans with type 2 diabetes. Health Psychol 2005 Jul;24(4):349-357. [CrossRef] [Medline]
    51. Bhattacharya G. Self-management of type 2 diabetes among African Americans in the Arkansas Delta: a strengths perspective in social-cultural context. J Health Care Poor Underserved 2012 Feb;23(1):161-178. [CrossRef] [Medline]
    52. Grant RW, Devita NG, Singer DE, Meigs JB. Improving adherence and reducing medication discrepancies in patients with diabetes. Ann Pharmacother 2003;37(7-8):962-969. [Medline]
    53. Piette JD, Heisler M, Harand A, Juip M. Beliefs about prescription medications among patients with diabetes: variation across racial groups and influences on cost-related medication underuse. J Health Care Poor Underserved 2010 Feb;21(1):349-361. [CrossRef] [Medline]
    54. Dunning T, Manias E. Medication knowledge and self-management by people with type 2 diabetes. Aust J Adv Nurs 2005;23(1):7-14. [Medline]
    55. Chao J, Nau DP, Aikens JE, Taylor SD. The mediating role of health beliefs in the relationship between depressive symptoms and medication adherence in persons with diabetes. Res Social Adm Pharm 2005 Dec;1(4):508-525. [CrossRef] [Medline]
    56. Funnell MM. The Diabetes Attitudes, Wishes, and Needs (DAWN) Study. Clinical Diabetes 2006 Oct 01;24(4):154-155. [CrossRef]
    57. Walker EA, Molitch M, Kramer MK, Kahn S, Ma Y, Edelstein S, et al. Adherence to preventive medications: predictors and outcomes in the Diabetes Prevention Program. Diabetes Care 2006 Sep;29(9):1997-2002 [FREE Full text] [CrossRef] [Medline]
    58. Davies MJ, Gagliardino JJ, Gray LJ, Khunti K, Mohan V, Hughes R. Real-world factors affecting adherence to insulin therapy in patients with Type 1 or Type 2 diabetes mellitus: a systematic review. Diabet Med 2013 May;30(5):512-524. [CrossRef] [Medline]
    59. Tang TS, Brown MB, Funnell MM, Anderson RM. Social support, quality of life, and self-care behaviors amongAfrican Americans with type 2 diabetes. Diabetes Educ 2008;34(2):266-276. [CrossRef] [Medline]
    60. Mayberry LS, Osborn CY. Family support, medication adherence, and glycemic control among adults with type 2 diabetes. Diabetes Care 2012 Jun;35(6):1239-1245 [FREE Full text] [CrossRef] [Medline]
    61. Henry SL, Rook KS, Stephens MA, Franks MM. Spousal undermining of older diabetic patients' disease management. J Health Psychol 2013 Dec;18(12):1550-1561 [FREE Full text] [CrossRef] [Medline]
    62. Mayberry LS, Harper KJ, Osborn CY. Family behaviors and type 2 diabetes: What to target and how to address in interventions for adults with low socioeconomic status. Chronic Illn 2016 Apr 19;12(3):199-215. [CrossRef] [Medline]
    63. Brod M, Christensen T, Kongsø JH, Bushnell DM. Examining and interpreting responsiveness of the Diabetes Medication Satisfaction measure. J Med Econ 2009;12(4):309-316. [CrossRef] [Medline]
    64. Odegard PS, Capoccia K. Medication taking and diabetes: a systematic review of the literature. Diabetes Educ 2007;33(6):1014-29; discussion 1030. [CrossRef] [Medline]
    65. Carter-Edwards L, Skelly AH, Cagle CS, Appel SJ. “They care but don't understand”: family support of African American women with type 2 diabetes. Diabetes Educ 2004;30(3):493-501. [Medline]
    66. Chlebowy DO, Hood S, LaJoie AS. Facilitators and barriers to self-management of type 2 diabetes among urban African American adults: focus group findings. Diabetes Educ 2010;36(6):897-905. [CrossRef] [Medline]
    67. Nelson LA. An SMS and IVR intervention improves medication adherence among adults with T2DM and low SES. 2016 Presented at: 37th Society of Behavioral Medicine's Annual Meeting & Scientific Sessions; April 1, 2016; Washington, DC. [CrossRef]
    68. Harris PA. Research Electronic Data Capture (REDCap) - planning, collecting and managing data for clinical and translational research. BMC Bioinformatics 2012;13(Suppl 12):A15. [CrossRef]
    69. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009 Apr;42(2):377-381 [FREE Full text] [CrossRef] [Medline]
    70. Callahan CM, Unverzagt FW, Hui SL, Perkins AJ, Hendrie HC. Six-item screener to identify cognitive impairment among potential subjects for clinical research. Med Care 2002 Sep;40(9):771-781. [CrossRef] [Medline]
    71. Dick JJ, Nundy S, Solomon MC, Bishop KN, Chin MH, Peek ME. Feasibility and usability of a text message-based program for diabetes self-management in an urban African-American population. J Diabetes Sci Technol 2011 Sep;5(5):1246-1254 [FREE Full text] [Medline]
    72. Lyles CR, Sarkar U, Osborn CY. Getting a technology-based diabetes intervention ready for prime time: a review of usability testing studies. Curr Diab Rep 2014 Oct;14(10):534. [CrossRef] [Medline]
    73. Georgsson M, Staggers N. An evaluation of patients' experienced usability of a diabetes mHealth system using a multi-method approach. J Biomed Inform 2016 Feb;59:115-129 [FREE Full text] [CrossRef] [Medline]
    74. Nielsen J. Nielsen Norman Group. 2000. Why You Only Need to Test with 5 Users   URL: https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ [accessed 2016-01-19] [WebCite Cache]
    75. Nielsen J. Nielsen Norman Group. 2006. Quantitative Studies: How Many Users to Test?   URL: https://www.nngroup.com/articles/quantitative-studies-how-many-users/ [accessed 2016-01-20] [WebCite Cache]


    Abbreviations

    EHR: electronic health record
    FQHC: Federally Qualified Health Center
    HbA1c: hemoglobin A1c
    HIPAA: Health Insurance Portability and Accountability Act
    IMB: information-motivation-behavioral skills
    IVR: interactive voice response
    MED: MEssaging for Diabetes
    REACH: Rapid Education/Encouragement And Communications for Health
    SES: socioeconomic status
    SMBG: self-monitoring of blood glucose
    SMS: short message service
    T2DM: type 2 diabetes mellitus


    Edited by G Eysenbach; submitted 27.05.16; peer-reviewed by I Ricci-Cabello, C Lyles PhD, K Nicholson; comments to author 23.06.16; revised version received 11.08.16; accepted 21.08.16; published 08.09.16

    ©Lyndsay A Nelson, Lindsay S Mayberry, Kenneth Wallston, Sunil Kripalani, Erin M Bergner, Chandra Y Osborn. Originally published in JMIR Human Factors (http://humanfactors.jmir.org), 08.09.2016.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on http://humanfactors.jmir.org, as well as this copyright and license information must be included.