A Patient-Facing Diabetes Dashboard Embedded in a Patient Web Portal: Design Sprint and Usability Testing

Introduction

Background

Diabetes is a leading cause of kidney failure, heart disease, stroke, visual impairment, and nontraumatic lower limb amputations [1]. Many of these complications can be delayed or prevented through disease control. Research demonstrates that diabetes self-monitoring, preventative health services, medication adherence, regular exercise, and attention to diet can lead to improved outcomes [2,3]. Despite their importance, few patients consistently receive all recommended services or engage in recommended self-care behaviors that can be challenging to implement and sustain [4,5]. Many patients with diabetes struggle with the knowledge and motivation necessary to successfully manage their disease [6].

Interventions aimed at enhancing patients’ motivation, skills, knowledge, and confidence in diabetes self-care have had limited success, with many relying on face-to-face interactions that are costly and challenging to scale [7,8]. Web-based diabetes self-management interventions have the potential to overcome these limitations; however, these interventions have also demonstrated variable effects on patients’ self-care and glycemic control [9,10]. Mixed results have been attributed to differences in the design and usability of these Web-based interventions, leading to varying degrees of user engagement [10,11]. Web-based interventions with greater user engagement are associated with better outcomes [12,13]. However, some Web-based interventions have not involved end users in the design process [14,15], and many have failed to include one or more recommended features for increasing patient engagement, including (1) ability to track, visualize, and summarize health data; (2) guidance in response to the data displayed; (3) ability to communicate with health care providers; (4) peer support; and (5) motivational challenges using elements of game design and competition [11,16].

Human-centered design is an approach to software development that emphasizes optimal user experience by integrating users directly into the design process and helps ensure the creation of a suitable user interface [17,18]. One human-centered design method, called design sprint, is a rapid 5-phase user-centered process that utilizes design principles to understand the problem, explore creative solutions, identify and map the best ideas, prototype, and ultimately test [17,18]. Usability testing ensures that Web-based interventions meet users’ expectations and work as intended, such that users are able to efficiently and effectively interact with the website [11]. Although usability testing is sometimes performed once the Web-based intervention has been fully developed, incorporating usability testing into the design process beginning with the earliest prototype provides the greatest opportunity to inform and improve the user interface design [17,18].

Objectives

This paper describes the application of design sprint methodology paired with mixed-methods, task-based usability testing to design and evaluate an innovative, patient-facing diabetes dashboard embedded in an existing patient portal, My Health at Vanderbilt (MHAV) [19] and integrated into an electronic health record. In particular, we sought to design a dashboard that addresses the needs of users, allows users to easily comprehend their diabetes health data, incorporates recommended strategies for increasing user engagement, and is satisfying and easy to use.

Methods

Dashboard Design

We utilized a 5-day design sprint methodology [17,18] developed by Google Ventures (Alphabet Inc, Mountain View, CA) to design our initial dashboard prototype. The process was facilitated by an experienced health information technology expert (ALT) who specializes in user experience (UX) and product design. A 5-day design sprint approach was chosen over other iterative agile methodologies because a design sprint approach offered the ability to rapidly develop a user-centered solution in the form of a prototype that could be tested and revised before investing limited research funds into the programming of the dashboard.

On day 1, we began by mapping out our challenge (Figure 1) to create a dashboard that would satisfy patients’ desire for information regarding their diabetes health status and address existing challenges in patients’ diabetes knowledge and motivation for diabetes self-management [5,20]. This process was informed by a review of the literature [14,21-30] from which we identified factors contributing to the limited efficacy of existing digital interventions, including (1) absence of user-centered design [14], (2) lack of integration with the health care delivery system [22,28], (3) absence of key features to maximize patient engagement, including patient-centered motivational strategies [29], and (4) failure to account for the unique needs of older patients and those with limited health literacy [30-32]. In addition, we reviewed recommended strategies to increase patient activation [6,33] (ie, the motivation, knowledge, skills, and confidence for managing one’s health condition) using mobile apps [16] and prior research on the potential role of social comparison information for motivating diabetes self-care [27,34].

‎View this figure

We also met one-on-one with expert stakeholders (eg, patient portal users with diabetes, diabetes educators, behavioral scientists, physicians, educators, and nurses) to ask questions aimed at enhancing our understanding of the challenge and refine our map. We identified expert stakeholders by approaching organizational leaders with a description of the project and by asking them to identify individuals in their area who could provide valuable input. For example, we approached the director of the Vanderbilt University Hospital Patient and Family Advisory Council who connected us with patients from the Council, who had diabetes, were current patient portal users, and expressed interest in improving care for people with diabetes. Experts’ comments were recorded in the form of how might we (HMW) statements [17,18]. The HMW method is used in design thinking to take insights and challenges and reframe them as opportunities [17,18]. Consistent with design sprint methodology, experts’ HMW statements were reviewed (Figure 2) to identify statements that shared a common theme. This was followed by grouping the statements into categories based on emerging themes to identify the most useful ideas for building the prototype. Experts encouraged the authors to consider how we might design the dashboard to (1) maximize accessibility, (2) frame diabetes health data in ways that promote patients’ understanding and motivate health behaviors, (3) facilitate patient action in response to the data they see (eg, patient resources and referral services), (4) enable communication with their health care team, (5) enhance social supports, and (6) incorporate strategies (eg, goal setting, progress tracking, and positive reinforcement) that motivate health behavior and keep users engaged.

On day 2, the existing ideas, architecture, and designs from health care and other industries related to the challenge were reviewed to establish the building blocks of our prototype. For example, existing solutions for displaying health and performance data and other types of quantitative, longitudinal, and benchmarked data from other industries (eg, finance and education) were reviewed. Subsequently, findings from the review and the meetings with expert stakeholders were used to sketch our own solutions (Figure 3).

On day 3, the solutions were critiqued and the solutions that had the greatest potential to successfully meet the challenge in the long term were decided by consensus. Following this, the authors adapted the solutions chosen to create a storyboard or step-by-step plan for the prototype (Figure 4).

On day 4, the authors developed the prototype using Apple Keynote (Apple Inc, Cupertino, CA) [35]. They collected assets (eg, stock imagery or icons) and stitched all components of the prototype together. Keynote slides (ie, screens) were tethered together using the animate feature to transition from one slide (ie, screen) to the next based on the action the user performs within the prototype. This resulted in an initial prototype (Figure 5) that functioned similar to a real webpage and was ready for the first round of usability testing on day 5. The initial prototype displayed and summarized 5 measures of patients’ diabetes health status (ie, hemoglobin A_1c [HbA_1c], systolic blood pressure, low-density lipoprotein cholesterol, microalbumin, and flu vaccination status). The existing literature on patient’s information needs when interpreting test results and strategies for improving comprehension was reviewed [36-38]. In addition, the authors identified recommended strategies for using patient-facing technologies to increase patient activation and incorporated dashboard functionality that matched each strategy. For example, for each measure, the dashboard used graphics to visualize and summarize health data and reinforce understanding with a color-coded system (red, yellow, and green) similar to the National Heart, Lung, and Blood Institute’s asthma treatment guideline [39] to indicate when action is needed. To facilitate understanding, we paired each measure with hyperlinks to literacy level–appropriate educational materials. To help motivate patients, the dashboard provided patients with social and goal-based comparison information regarding their diabetes health status [27,34]. In addition, using elements of game design, a star rating provided patients with feedback on the number of measures at goal. To facilitate communication with their health care team, patients could click a link to contact their doctor’s office via a secure message. Reminders for self-care (eg, take medication, exercise, etc) could be set and delivered to patients’ mobile phones or email, and diabetes self-care goals could be set and tracked.

Usability Study Design

From September to October 2016, we conducted a mixed-methods, task-based usability study of dashboard prototypes with individual patients under controlled conditions. Patients were recruited from the Vanderbilt Adult Primary Care (VAPC) clinic. Individual usability sessions lasted between 30 and 75 min. Given that the majority of usability problems are commonly identified within the first 5 usability evaluations [40-42], each round of usability testing included between 3 and 6 participants. After each round of usability testing, the dashboard prototype was revised in response to usability findings before the next round of testing.

‎View this figure

‎View this figure

‎View this figure

‎View this figure

Setting

The VAPC clinic is located within the Vanderbilt University Medical Center (VUMC) in Nashville, TN. The clinic cares for about 25,000 unique patients annually, of which about 4500 (18.00%) have diabetes. All clinical data are entered into an electronic health record, and the patients are provided access to their clinical data via a Web portal.

Participants and Recruitment

Participants were eligible for the study if they had type 2 diabetes mellitus, were English-speaking, were aged 21 years or older, and were current users of the VUMC patient Web portal, MHAV. Potential participants were identified automatically using VUMC’s Subject Locator to query the electronic health records of patients with upcoming clinic appointments for discrete inclusion and exclusion criteria. Identified patients (n=334) were mailed a letter describing the study and asked to contact the investigators if they were interested in participating. Interested patients (n=22) contacted the research coordinator to learn more about the study and confirm eligibility. Patients who agreed to participate (n=17) were scheduled to participate in a usability session on the day of their clinic appointment. Overall, 3 patients canceled due to weather or a conflicting appointment. A total of 14 patients ultimately completed a usability session and provided written informed consent before participating in their session. The Vanderbilt University Institutional Review Board approved this research.

Data Collection and Measures

Before the usability testing session, enrolled patients were asked to complete a short questionnaire before their interview. The questionnaire included basic demographic questions, including items about computer and smartphone usage and internet access, as well as validated measures of health literacy [43] and numeracy [44]. In addition, data regarding comorbidities were extracted from participants’ medical record as reported by the physicians within the patients’ problem list.

Each participant received a standardized introduction to the dashboard and the think-aloud procedure that allows testing observers to understand and track a participant’s thought processes as they navigate the dashboard [45]. One of the authors (ALT) led each session using a semistructured interview guide, while another author (WM) observed and took notes. With a dashboard prototype that contained fictitious patient data, participants were asked to perform common standardized tasks including logging in, retrieving HbA_1c data, messaging their doctor, setting a reminder, and setting a goal. The tasks were designed to represent what typical users might do when visiting their dashboard. All participants accessed and navigated the dashboard using a 15-inch MacBook Pro 11,3 (2014 generation) with an external mouse and Chrome Web browser with default resolution. In addition, after participants attempted each assigned task (eg, message your doctor), the interviewers used open-ended questions outlined in the interview guide to elicit participants’ (1) expectations for the feature’s functionality, (2) ability to comprehend the information displayed, (3) ability to navigate to and from the feature, (4) satisfaction with the feature, and (5) how the feature might be improved. Each session was audio-recorded, and the computer screen was video-recorded using QuickTime Player (Apple Inc, Cupertino, CA).

To assess and quantify participant satisfaction with the dashboard, at the conclusion of their usability session, participants completed 12 items from the Computer System Usability Questionnaire (CSUQ), which assess participants’ perceptions of the dashboard’s ease of use, likability of the interface, and overall satisfaction using a 7-point Likert response scale (1=strongly disagree to 7=strongly agree), with 7 indicating the highest possible satisfaction [46].

Data Analysis

Task Completion Analysis

Task completion was coded with a usability rating scale utilized in prior studies [47-49]. Task completion was rated on a 5-category scale: (1) successful/straightforward, (2) successful/prolonged, (3) partial, (4) unsuccessful/prolonged, and (5) gave up [47]. Two coders first coded the same usability session video (not used in the analysis) to calibrate their coding. They subsequently coded the remaining videos independently. Disagreements were resolved by consensus, and both coders were blinded to the dashboard prototype representing the initial prototype and the prototypes that were revisions.

Interview Analysis

Audio files of interviews were submitted to a professional transcription service, Rev.com Inc (San Francisco, CA). Transcripts were checked for accuracy and identifying information was removed. Deidentified transcripts were imported into NVivo 10 (version 10; QSR International, Burlington, VT) for coding and analysis. Similar to other health app usability studies [47,50], we used selective coding to capture participants’ comments about usability concerns [51]. Participant comments were sorted into categories that addressed 3 elements of usability: design, efficiency of use, and content and terminology [52]. A research assistant with training in qualitative methods coded all interviews. After the initial coding, a second trained coder reviewed each code and noted any discrepancies. The 2 coders then met and resolved any differences by consensus. Illustrative quotes from participants were edited slightly for grammar and clarity for inclusion in this paper. Participants’ comments informed revisions to the dashboard prototype.

Statistical Analysis

Descriptive statistics were used to characterize the study participants, task completion, and survey data. All analyses were completed with SAS version 9.4 (SAS Institute, Inc, Cary, NC).

Stop Criteria

Data analysis began after the initial round of testing, and the authors used the findings to inform prototype revisions before the subsequent round of testing. Additional rounds of testing were conducted until the majority of participants within a round of testing (1) were able to successfully complete all tasks, (2) indicated high overall satisfaction with the dashboard as assessed by the overall satisfaction item on the CSUQ (score≥6), and (3) expressed no new usability concerns during the interview (ie, saturation).

Results

Participants

Table 1 shows participant characteristics. The sample (N=14) comprised 5 patients in round 1, 3 patients in round 2, and 6 patients in round 3; at this point, the authors reached their stop criteria. Participants’ mean age was 63 years (range 45-78 years), 57% (8/14) were female, and 50% (7/14) were white. All participants reported using a home computer, and 64% (9/14) reported using a smartphone. All participants had home internet access. Most participants had one or more comorbid diseases in addition to diabetes.

Task-Based Usability

Figure 6 illustrates task performance among the 5 participants in round 1 who tested the initial prototype compared with the 6 participants in round 3 who tested the final prototype. Participants attempted 5 tasks that ranged in complexity from logging in to setting a reminder.

Tasks: (A) Log-In and (B) Set a Goal

All participants in both rounds straightforwardly logged in to the dashboard and set a goal.

Task: (C) Identify Most Recent Hemoglobin A _1c

Only one participant in the initial round of testing was able to identify their most recent HbA_1c value from the dashboard. Most participants had difficulty interpreting the dial display, were confused regarding which icon on the dial indicated the user’s most current value, and could not comprehend the HbA_1c data. In response, the authors revised the data display design and status indicator icons. They relocated the features aimed at facilitating patients’ understanding of their health data, including a hover over info icon providing a nontechnical description of the measure (eg, HbA_1c) and links to literacy level–sensitive educational materials so they were adjacent to the data (see Figure 1 initial prototype and Figure 7 final prototype). After revisions, all 6 participants in the final round were able to complete the task and comprehend their data.

Task: (D) Message Doctor’s Office

All 5 participants in the initial round were able to message their doctor’s office; however, 2 participants hesitated or demonstrated some confusion despite completing the task. Participants indicated that they were accustomed to using the existing messaging icon within the header of the patient portal, and some struggled to locate the messaging icon within the dashboard. After revising the icon in response to feedback (ie, larger text, adding color and a button icon), the majority of participants in the final round successfully completed the task. However, 3 participants continued to initially attempt messaging via the existing icon in the header, one of whom completed the task only after being directed to the correct button icon.

‎View this figure

‎View this figure

Task: (E) Set a Reminder

Only 2 participants in round 1 were able to set a reminder on the dashboard. Participants struggled to set the frequency of recurrence and a stop date for reminders they wished to receive only for a specified time. Subsequently, the authors revised the layout of the “set reminder” pop up window to include a clear start and stop date and time, as well as a drop-down menu to set recurrences (eg, daily, weekly, etc). After revisions, 4 of 6 participants in round 3 were able to set a reminder, with one additional participant successfully completing the task with prolonged effort.

Participant Interviews

Table 2 shows the participants’ comments about usability concerns grouped by usability area. Several revisions were made in response to participants’ usability concerns, including revisions to the display of patients’ health data and star status, icons indicating the patient’s value and “patients like me” value, standardizing educational links and adding diet information, grouping and standardizing action items, enlarging the font size, and providing a frequently asked questions page (see Figure 1 initial prototype and Figure 7 final prototype).

Satisfaction Survey

Table 3 reports mean scores for the CSUQ items among participants in round 1 who tested the initial prototype compared with participants in round 3 who tested the final prototype. Participants who tested the initial prototype and those who tested the final prototype rated the usability above average (ie, scores >4 on a 7-point scale) for all 12 items. The mean score for all 12 items improved between the initial and final prototypes.

Discussion

Principal Findings

Our study illustrates the use of design sprint methodology alongside mixed-methods, task-based usability testing in the design of a Web-based intervention for patients with diabetes. By using this design approach, we were able to rapidly create a prototype and rigorously assess task-based usability before any programming. Task-based usability testing and qualitative analysis of interviews with a small number of participants quickly identified usability challenges that led to improvements in successive iterations. Participant feedback informed changes in the data display that led to improved comprehension of diabetes health data. Participants’ usability satisfaction surveys demonstrated a high level of satisfaction with the dashboard that improved from initial to final prototype. The final prototype incorporated recommended strategies to enhance patient activation across the engagement spectrum, from providing educational resources to promoting behavior change through rewards (see Figure 8) [16].

Building Upon Prior Research

Several prior studies have reported the design and usability of patient-facing health apps and Web-based interventions for patients with diabetes [50,53-58]. Approaches to the design of these health apps and Web-based interventions typically employ some variation of user-centered design [56-59]. A significant limitation of prior design approaches is the time and cost involved with the rapidly evolving pace of technology [60,61]. This study is the first in our knowledge to report the design of a digital health intervention using design sprint methodology and demonstrate its utility in efficiently and effectively designing a Web-based intervention that is satisfying to use.

By utilizing design sprint methodology, we were able to create a viable initial prototype within 5 days. Given the rapidly evolving technology and patient expectations of health technology [60,62], efficient yet rigorous design methodology is essential. We were able to enhance the scientific rigor of the design sprint approach by using validated measures of usability [46] and task-performance [47-49], as well as an established qualitative methodology to analyze interviews and determine saturation [51]. This approach allows usability concerns to be identified before programming, potentially saving the researcher both time and money. Consistent with the findings of Nielsen, we found that the majority of usability problems were identified in the first 5 usability evaluations, with diminishing returns after the eighth evaluation [40-42]. While enrolling additional participants in our study may have revealed additional usability concerns, our sample was sufficient to establish a minimally viable product (eg, final prototype) that allowed us to proceed to program the dashboard with the reasonable confidence that most usability issues were identified and addressed. As with any app or website, ongoing attention to user feedback and iterative improvements are likely to continue indefinitely as technology and users evolve. Although some usability studies employ a large number of participants, this is mostly done to provide sufficient sample size for quantitative analyses, and additional participants yield relatively few new usability concerns [40-42]. In addition, our usability findings build upon other recent studies of patient-facing diabetes health apps [50,53,59]. Georgsson et al used a similar mixed-methods approach to evaluate the usability of their mHealth system for diabetes type 2 self-management [53]. Similar to this study, their study included task-based testing with a think-aloud protocol, semistructured interviews, and a questionnaire on patients’ experiences using their system. Consistent with Georgsson et al, we found a mixed-methods approach resulted in a comprehensive understanding of usability. Our study extends these findings by demonstrating the effectiveness of this approach to objectively assess and track usability in response to iterative revisions of a prototype in the design phase.

‎View this figure

Our study also has implications for the design of patient portals and the display of patients’ health data. By giving patients direct access to their health data, patient portals can improve patient engagement [63] and empower patients to actively participate in their care [64]. However, research suggests that patients struggle to understand health data communicated to them via patient portals [65]. A recent study by Giardian et al suggests that current patient portals do not display health data in a patient-centered way, which can lead to misunderstandings and patient distress [66]. In our study, patients had difficulty comprehending HbA_1c data in the dial display (Figure 1) that improved with ruler display (Figure 7), demonstrating the importance of user-centered design. Although the content was relatively unchanged, we revised the display based on user feedback, resulting in increased comprehension and improved visibility of features aimed at facilitating patients’ understanding of their health data.

Limitations

This study has important limitations. We recruited a convenience sample of patients from a single, large, urban academic medical center that may limit the generalizability of our findings. Our sample included patients who were more educated and had greater computer and internet access than the overall population of patients with diabetes [67,68]. For future studies, researchers should consider purposive sampling to recruit patients with specific characteristics. Given the known barriers to usability among older patients [15], a strength of our sample was the inclusion of a majority of patients over the age of 60 years that allowed us to ensure the dashboard usability among this demographic. In addition, although we were able to directly observe individual users as they attempted several assigned tasks using the dashboard, our data are subject to the Hawthorne effect (ie, altered behavior due to an awareness of being observed). Similarly, we did not collect data on how patients would engage with the dashboard on their own. It would be useful to collect actual-use data in future studies including the level of engagement with specific dashboard functions over time. Although we designed the dashboard with elements aimed at increasing patient activation, this study focused on the design and task-based usability of the dashboard and not on the evaluation of its impact. Further research is needed to test the efficacy of the dashboard on cognitive, behavioral, and clinical outcomes including patient activation.

Researchers and others considering using design sprint methodology should also consider some of the limitations of the approach. Although a standard design sprint that unfolds over 5 days is generally recommended [17,18], researchers may wish to experiment with shorter, or more likely, longer sprints. Design sprint methodology relies on understanding the user (ie, the consumer and their needs), and in some instances, it may be necessary to spend additional time before the design sprint to understand the target user and their needs and challenges. In our case, a literature review on the patients’ experiences with portal use, challenges with diabetes self-management, and the limitations of existing diabetes apps provided insights about our target users. Design sprints also rely heavily on the ideas generated from the solutions sketched by team members on day 2. Therefore, this phase of idea generation should not be shortened and may, in fact, benefit from more time.

Conclusions

In conclusion, the results underscore the value of design sprint methodology to efficiently create a viable user-centric prototype of a Web-based intervention and the importance of mixed-methods evaluation of usability as a part of the design phase beginning with the initial prototype. Design sprints offer an efficient way to define the problem, assess the needs of users, iteratively generate ideas and develop a viable product for testing, whereas usability evaluation methods ensure health apps and Web-based interventions appeal to users and support their use.