Diabetes MAP is a self-guided, Web-delivered intervention designed to promote medication adherence among patients with T2DM. Diabetes MAP’s content is grounded in the Information-Motivation-Behavioral skills (IMB) model of medication adherence [24,25]. Studies in diabetes [24,25] and other chronic disease contexts (eg, HIV) [26] suggest a patient’s medication adherence depends on his or her adherence-related information, motivation to take medications, and adherence-related behavioral skills. Therefore, Diabetes MAP’s intervention content addresses user-specific barriers to adherence in each of these domains. Upon registering for an account and logging in to the Diabetes MAP site, users are asked to create a medication list by searching for RxNorm-generated medications. Next, they are asked a series of questions to assess their medication adherence-related information, motivation, and behavioral skills barriers. Entered medications and responses to these questions populate a separate page titled, My Tailored Tools (Figure 1, Top Panel). This page responds to a user’s early inputs (eg, medications entered, barriers to adherence) with a toolbox of tailored regimen-specific and IMB-model based intervention content.

My Tailored Tools houses 30 educational videos and 11 pieces of static content to enhance user-specific adherence-related information, motivation, and behavioral skills. Informational content is both medication class-specific (eg, a video on the key facts about metformin and how it works in the body for patients prescribed metformin, a video of how insulin works in the body for patients prescribed insulin) and conveys the importance of adherence for glycemic control and preventing complications (eg, a video showing the complications that can occur from not taking medications as prescribed). Motivational content is intended to enhance patients’ personal and social motivation for adherence (eg, a video on how to overcome one’s fear of needles, static content presenting strategies for soliciting social support for adherence). Finally, behavioral skills content provides practical “how to” advice to ensure successful adherence (eg, a video with step-by-step instruction on how to inject insulin, a video on how to store insulin).

The Diabetes MAP website has additional capabilities. Its features (ie, functionality built into the site to enhance the user experience) allow users to perform various tasks (ie, clearly defined assignments to complete within a website). For example, upon creating a medication list, users can print and email this list, learn about each medication listed, set up medication dosing and refill reminders sent as text messages to their mobile phone (Figure 1, Bottom Panel), and connect to a patient portal account (ie, My Health at Vanderbilt) to communicate with healthcare providers about medication side effects and prescription reauthorizations via secure messaging. As noted above, users can also complete an IMB model-based barriers-to-adherence assessment and view user-specific educational videos and content to address users’ IMB model-based barriers. Finally, the website includes navigational videos explaining the site’s features and giving instructions on how to complete tasks. Diabetes MAP was not designed for a specific user, but we made design choices to account for potential literacy, visual, and auditory limitations of all users. Such choices include presenting simplified language in large font, the option to watch and/or listen to videos or read video scripts, and a full-screen option to improve video visibility.

To test the usability of the Diabetes MAP site, we recruited English-speaking adults from an academic medical center who were diagnosed with T2DM, prescribed diabetes medications, and had Internet access to participate in a mixed-methods study. Recruitment strategies included advertisements about the study, referrals from healthcare providers, medical center listserv announcements, and approaching patients waiting in an adult diabetes specialty clinic or adult primary care clinic. The total number of participants enrolled (N=32) reached the target enrollment for qualitative (at least 5) and quantitative (at least 20) usability testing [27,28]. The Institutional Review Board at Vanderbilt University Medical Center approved all study procedures prior to participant enrollment.

A trained research assistant (RA) scheduled interested and eligible participants to meet individually in a private room at the medical center. The RA administered (1) informed consent, (2) a survey by reading survey items and response options out loud or by distributing one available in paper-pencil format or electronic format via Research Data Capture (REDCap™) [29] that could be completed independently, and (3) a 1-page instruction guide on how to locate and access the Diabetes MAP website, which each participant was asked to independently use for 2 weeks. With permission, the RA also reviewed each participant’s medical record to collect clinical data.

We used Mouseflow ApS™ to measure participants’ use of Diabetes MAP. After 2 weeks, the RA invited participants to provide feedback on the site’s usability by completing a 20-minute survey and attending a 60-minute focus group session. The survey could be completed in REDCap™ via an email link or in-person immediately before the focus group session. A trained focus group facilitator used semistructured a priori questions to assess participants’ experiences with Diabetes MAP, demonstrate the site on a projector screen, and elicit responses and impressions of the site. This method is consistent with the pluralistic walkthrough approach to usability testing that involves stepping through a system with users to understand their perceptions of and experiences with a system [30,31]. The pluralistic walkthrough approach reveals users’ uncertainty with a system’s features and tasks better than other usability methods [31]. We asked participants if they had challenges with using the site, their most and least favorite aspects of the site, their perceived benefits of using the site, and any recommendations they had for improving it. All sessions were audio-recorded. Recordings were transcribed verbatim and de-identified prior to analyses. We compensated participants up to $155 for completing a survey at enrollment ($25), using Diabetes MAP ($8 per hour, up to 10 hours), completing the follow-up survey ($15), and participating in a focus group ($35).

We used SPSS version 21.0 to summarize quantitative data using means and standard deviations (SD), or frequencies and percentages as appropriate. We used selective coding [33] to identify focus group comments and conversations [34] addressing the 6 areas of website usability: (1) intuitive design, (2) ease of learning, (3) efficiency of use, (4) error frequency and severity, (5) memorability, and (6) subjective satisfaction [22]. First, we read focus group transcripts in their entirety, highlighting participant comments related to opinions about, experiences with, and suggestions for Diabetes MAP. Next, we integrated similar comments into categories. After an iterative process of integration and refinement, we mapped categories of responses onto each usability area. Units of analysis included single participant comments and multi-participant conversations reporting similar or different experiences with the site and suggestions for improving the Diabetes MAP user experience (eg, strategic placement of instructions, features and Web content).

On the survey, participants rated Diabetes MAP’s usability above average (ie, scores of >3 on a 5-point scale) on each of the 10 items (Table 2). The total usability rating averaged across all items was 3.86 ± 0.90. We also mapped each survey item onto the usability area it most closely reflected. As shown in Table 2, participants rated the understandability of the site’s information (ie, intuitive design), the clarity of the site’s information (ie, intuitive design), and the pleasantness of interacting with the site (ie, subjective satisfaction) the most favorably. In contrast, participants rated the ease and quickness of recovering from mistakes (ie, error frequency and severity), the effectiveness of the site’s information in helping users complete tasks (ie, efficiency of use), and the ease of navigating the site (ie, intuitive design) the least favorably (Table 2).

Across 9 focus groups, participants shared experiences using Diabetes MAP, including concerns about its usability and recommendations for improvement. Of the 24 unique usability concerns reported, 14 concerns were mentioned in the first focus group and another 6 concerns were mentioned in the second focus group. By the fourth focus group, 95% of all unique usability concerns had been reported. Generally, participants’ experiences with Diabetes MAP were similar across age, gender, race/ethnicity, education, and income. Table 3 presents each unique concern organized by usability area [22] and the number of focus groups it was mentioned in. The 6 areas of website usability [22] provide a framework for examining participants’ usability concerns and recommendations for improvement.

When participants voiced concerns about Diabetes MAP, they also gave suggestions for improving it. Common across all suggestions was a request for more simplicity and flexibility within the site. To improve the site’s ease of learning and efficiency of use, participants wanted more straightforward methods for accessing the site’s components. They suggested strategically placing instructions, features, and Web content in easily recognizable ways. Some participants wanted specific features to be accessible on every page, or clearly designated on their own page. For example, participants liked the idea of a page dedicated to setting up text message reminders. On this page, previously entered medication information (ie, name, dosage) would appear and users could set up reminders for when to take specific medications and order prescription refills.

To improve the site’s intuitive design, participants recommended the navigational videos be minimized or eliminated entirely.

Additionally, in reference to the site’s memorability, participants recommended using a simple and recognizable URL that is easy to locate with an online search. They also wanted Diabetes MAP’s purpose to be clear while using it (eg, spelling out the Diabetes MAP acronym and including images of diabetes medications throughout the website).

Participants wanted a simplified, streamlined Diabetes MAP user experience. In order to improve the site’s error frequency and severity, participants recommended increasing compatibility across multiple browsers, including older versions of commonly used Web browsers. Additionally, they requested the ability to use Diabetes MAP across multiple digital devices without loading time delays. They also stressed the importance of clear and accessible resources for user support (eg, the ability to contact study staff directly if they had issues or questions, an accessible and searchable help resource on the website itself). Finally, participants wanted functional, useful, and easy to recall navigational videos to further facilitate learning how to use the site.

Those who reported some dissatisfaction, but generally endorsed the utility of Diabetes MAP, felt it might be more appropriate for certain types of patients with diabetes. For example, some users diagnosed with diabetes for a longer period of time felt the website might be particularly helpful for newly diagnosed patients. Other users suggested the website might be more useful for younger, more technology-proficient patients who prefer technology-delivered information as opposed to more traditional print materials.

Usability testing is issue-focused and designed to assess the extent to which users can easily, efficiently, and effectively perform tasks with a technical system. We employed a mixed-methods approach to understand the challenges of using a Web-delivered medication adherence promotion intervention called Diabetes MAP. Participants with diabetes provided ratings and descriptions of their experiences using the website, as well as recommendations for improving it. On surveys, participants agreed Diabetes MAP was helpful and easy to use, but, in focus groups, they mentioned 24 unique user concerns related to each of the 6 factors determining website usability [22].

Our quantitative results are comparable to other usability studies employing the CSUQ, in that total ratings were above average [36,37]. When comparing survey items rated most to least favorably with focus group comments, there were instances when survey ratings and quotes were discordant and concordant. For example, participants rated the understandability and clarity of the site’s information and the pleasantness of interacting with the site most favorably on surveys, yet, in focus groups, several participants expressed frustration with understanding how to complete tasks and navigate the website (ie, issues with the site’s ease of learning and intuitive design). Alternatively, many positive statements about the value of the site’s information and features (ie, subjective satisfaction) support these high ratings. Participants rated the ease and quickness of recovering from mistakes least favorably on surveys; focus group comments about error frequency and severity align with this low rating. In reconciling these inconsistencies and consistencies, it appears that while some participants had issues with understanding, navigating, and accessing Diabetes MAP, participants who successfully accessed the site, said it was enjoyable and helpful.

Recent usability studies of Web-delivered interventions for T2DM self- management yield results comparable to ours. In their evaluation of a Web-based dietary intervention, Ramadas et al found positive ratings of a website’s usability based on survey items; however, this study did not use qualitative assessments [38]. Alternatively, Yu et al used focus groups to examine the usability of their self-management website [39] and identified several of the same usability concerns we did with Diabetes MAP. Namely, participants mentioned issues with the website layout and organization, navigation, data entry, and language [39]. Although these issues can be applied to Web-delivered interventions generally, usability testing also reveals issues specific to a certain website [39].

Our research highlights the value of using mixed methods for usability testing. Had we relied on only survey data, we would have incomplete information on Diabetes MAP’s usability. Collecting qualitative data as part of usability testing reveals insights on unanticipated challenges and ideas for improving a site [21]. Additionally, involving members from the target audience is critical to understanding any unique needs of users for whom the site is intended [40]. The total time logged into Diabetes MAP during the 2-week period varied considerably across users. In a similar usability study, Heinrich et al had participants use a diabetes education site for 2 weeks, and participants visited the site an average of 3.6 ± 2.7 times and spent an average of 58.0 ± 56.1 total minutes on the site [41]. Compensation for time spent on the site was not reported. In our study, the more time spent using Diabetes MAP may reflect compensating participants per hour of use. Despite this, our qualitative results suggest some participants were discouraged from logging in more often because of the usability issues they encountered.

The US Department of Health and Human Services (HHS) has set forth peer-reviewed guidelines for improving the design and usability of websites [42]. Taking into consideration these guidelines and the results of our usability study, we identified key principles for website creation to promote an optimal user experience. First, it is crucial to employ both simplicity and clarity in Web design. Diabetes MAP users were often confused about how to complete tasks and navigate the site both because they encountered user errors and the layout was unintuitive. HHS suggests standardizing tasks to be performed in a similar way so tasks can be reliably repeated [42]. When requesting users to enter information, a standard entry format should be used across tasks (eg, drop-down boxes). Additionally, to account for working memory limitations, content from 1 page that need be remembered on other pages should carry over to those pages [42]. Finally, using simplified and familiar terminology for a URL and website features will minimize user confusion and frustration.

Our second principle is to design websites with the goal of keeping users informed and aware of website processes. As a general observation from our focus groups, users became frustrated with unanticipated incidents (eg, long downloading times, automatically being logged out). In some cases, it may not be possible to reduce the size of a page to minimize the time it takes for a webpage to load [42]. However, a website can notify users of the time required to download an image and/or supply progress indicators (eg, an hourglass) to communicate a waiting period and its duration [42]. In either case, the user expects additional time instead of wondering how long to wait. HHS also recommends warning users if a page is going to “time out,” so they can request extra time if needed [42]. Websites should also provide assistance to users who need additional help and ensure users are aware of this assistance. Resources should be easily accessible on the site, such as links providing more information about site content and a section for frequently asked questions [42].

There are limitations to our study. Because we recruited our sample from a single academic medical center, our findings may not generalize to other patient populations. However, our sample characteristics map onto the academic medical center’s patient population for whom Diabetes MAP was designed for. Additionally, although we were able to track time spent using Diabetes MAP, we were unable to track how participants used their Vanderbilt patient portal account because the two websites are not integrated. Considering the recent advancement of patient-provider communication in Web-delivered interventions, it will be valuable to track usage with this type of feature in future studies. Another limitation is the reliance on retrospective self-reports of users’ experiences with Diabetes MAP. Furthermore, the range from when a participant used Diabetes MAP to when he/she participated in a focus group session was 14-88 days. Other usability testing methods, such as think-aloud protocols, cognitive walk-throughs, and remote user testing facilitate real-time data collection of user interactions with a system [30]. No single usability evaluation method can capture all usability problems [43]. While think-aloud studies combat the limitations of retrospective studies, they are limited by an unnatural situation in which users may feel uncomfortable talking to themselves and the possibility many statements will be filtered (ie, not reflective of users’ actual experience) [44]. Although user feedback in our study was retrospective and the time between participants’ website use and focus group feedback ranged widely, we presented Diabetes MAP on a large projector screen and re-oriented participants to its pages, functions, and features to solicit real-time feedback and prompt recall of users’ experience. This method is similar to the pluralistic walkthrough approach that reveals users’ uncertainty with a system’s features and tasks better than other usability methods [31]. Additionally, the number of participants in each focus group allowed for the majority (over 80%) of concerns to be reported after only 2 focus group sessions. Moreover, the use of a mixed-methods approach provided a comprehensive evaluation of Diabetes MAP’s usability. Finally, because usability feedback is system-dependent and the current study is based on a specific website, our findings may not generalize to other Web-delivered interventions.

Our findings highlight the importance of evaluating a website’s usability prior to testing its efficacy. For Web-delivered interventions to be used as intended, researchers, Web designers, and developers must plan sufficient time to perform usability testing [45]. Ideally, they should begin with design thinking to allow for experimentation, creation and prototyping, and feedback and redesign prior to final product presentation [46]. An understanding of the target group’s needs paired with iterative idea generation and development helps ensure the final website appeals to users and supports their using it [46]. Usability evaluation methods, including user-driven approaches, ensure users can optimally engage with and benefit from Web-delivered interventions [15,47]. Without ensuring a website’s usability, critical design and functionality issues may go unnoticed and unaddressed, thereby preventing a site’s benefits from being realized [15].