This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on http://humanfactors.jmir.org, as well as this copyright and license information must be included.
Web-delivered interventions are a feasible approach to health promotion. However, if a website is poorly designed, difficult to navigate, and has technical bugs, it will not be used as intended. Usability testing prior to evaluating a website’s benefits can identify barriers to user engagement and maximize future use.
We developed a Web-delivered intervention called Diabetes Medication Adherence Promotion (Diabetes MAP) and used a mixed-methods approach to test its usability prior to evaluating its efficacy on medication adherence and glycemic control in a randomized controlled trial.
We recruited English-speaking adults with type 2 diabetes mellitus (T2DM) from an academic medical center who were prescribed diabetes medications. A trained research assistant administered a baseline survey, collected medical record information, and instructed participants on how to access Diabetes MAP. Participants were asked to use the site independently for 2 weeks and to provide survey and/or focus group feedback on their experience. We analyzed survey data descriptively and qualitative data thematically to identify participants’ favorable and unfavorable experiences, characterize usability concerns, and solicit recommendations for improving Diabetes MAP.
Enrolled participants (N=32) were an average of 51.7 ± 11.8 years old, 66% (21/32) female, 60% (19/32) non-Hispanic White, 88% (28/32) had more than 12 years of education, half had household incomes over $50,000, and 78% (25/32) were privately insured. Average duration of diagnosed diabetes was 7.8 ± 6.3 years, average A1c was 7.4 ± 2.0, and 38% (12/32) were prescribed insulin. Of enrolled participants, 91% (29/32) provided survey and/or focus group feedback about Diabetes MAP. On the survey, participants agreed website information was clear and easy to understand, but in focus groups they reported navigational challenges and difficulty overcoming user errors (eg, entering data in an unspecified format). Participants also reported difficulty accessing the site and, once accessed, using all of its features. Participants recommended improving the site’s user interface to facilitate quick, efficient access to all features and content.
Adults with T2DM rated the Diabetes MAP website favorably on surveys, but focus groups gave more in-depth feedback on the user experience (eg, difficulty accessing the site, maximizing all of the site’s features and content, and recovering from errors). Appropriate usability testing methods ensure Web-delivered interventions work as intended and any benefits are not diminished by usability challenges.
Among adults with type 2 diabetes (T2DM), approximately 1 in 3 do not take their medications as prescribed [
Web-delivered interventions have mixed effects on health behaviors [
The evaluation of a website’s usability is necessary before testing its potential efficacy on health behaviors and outcomes [
Usability testing ensures a Web-delivered intervention works as intended, so the target audience uses it to the degree needed to reap its potential benefits [
Diabetes MAP is a self-guided, Web-delivered intervention designed to promote medication adherence among patients with T2DM. Diabetes MAP’s content is grounded in the Information-Motivation-Behavioral skills (IMB) model of medication adherence [
My Tailored Tools houses 30 educational videos and 11 pieces of static content to enhance user-specific adherence-related information, motivation, and behavioral skills. Informational content is both medication class-specific (eg, a video on the key facts about metformin and how it works in the body for patients prescribed metformin, a video of how insulin works in the body for patients prescribed insulin) and conveys the importance of adherence for glycemic control and preventing complications (eg, a video showing the complications that can occur from not taking medications as prescribed). Motivational content is intended to enhance patients’ personal and social motivation for adherence (eg, a video on how to overcome one’s fear of needles, static content presenting strategies for soliciting social support for adherence). Finally, behavioral skills content provides practical “how to” advice to ensure successful adherence (eg, a video with step-by-step instruction on how to inject insulin, a video on how to store insulin).
The Diabetes MAP website has additional capabilities. Its features (ie, functionality built into the site to enhance the user experience) allow users to perform various tasks (ie, clearly defined assignments to complete within a website). For example, upon creating a medication list, users can print and email this list, learn about each medication listed, set up medication dosing and refill reminders sent as text messages to their mobile phone (Figure 1, Bottom Panel), and connect to a patient portal account (ie, My Health at Vanderbilt) to communicate with healthcare providers about medication side effects and prescription reauthorizations via secure messaging. As noted above, users can also complete an IMB model-based barriers-to-adherence assessment and view user-specific educational videos and content to address users’ IMB model-based barriers. Finally, the website includes navigational videos explaining the site’s features and giving instructions on how to complete tasks. Diabetes MAP was not designed for a specific user, but we made design choices to account for potential literacy, visual, and auditory limitations of all users. Such choices include presenting simplified language in large font, the option to watch and/or listen to videos or read video scripts, and a full-screen option to improve video visibility.
Top Panel: Diabetes MAP screenshot of the My Tailored Tools page presenting videos and content to address a user's barriers to adherence. Bottom Panel: Diabetes MAP screenshot of the page where a user can set up text message medication reminders.
To test the usability of the Diabetes MAP site, we recruited English-speaking adults from an academic medical center who were diagnosed with T2DM, prescribed diabetes medications, and had Internet access to participate in a mixed-methods study. Recruitment strategies included advertisements about the study, referrals from healthcare providers, medical center listserv announcements, and approaching patients waiting in an adult diabetes specialty clinic or adult primary care clinic. The total number of participants enrolled (N=32) reached the target enrollment for qualitative (at least 5) and quantitative (at least 20) usability testing [
A trained research assistant (RA) scheduled interested and eligible participants to meet individually in a private room at the medical center. The RA administered (1) informed consent, (2) a survey by reading survey items and response options out loud or by distributing one available in paper-pencil format or electronic format via Research Data Capture (REDCap™) [
We used Mouseflow ApS™ to measure participants’ use of Diabetes MAP. After 2 weeks, the RA invited participants to provide feedback on the site’s usability by completing a 20-minute survey and attending a 60-minute focus group session. The survey could be completed in REDCap™ via an email link or in-person immediately before the focus group session. A trained focus group facilitator used semistructured a priori questions to assess participants’ experiences with Diabetes MAP, demonstrate the site on a projector screen, and elicit responses and impressions of the site. This method is consistent with the pluralistic walkthrough approach to usability testing that involves stepping through a system with users to understand their perceptions of and experiences with a system [
Demographic information included participants’ age, gender, race/ethnicity, years of education, annual household income, and health insurance status. We asked participants whether they owned a mobile phone and used text messaging with their phone to better understand participants’ reasons for setting up or not setting up medication dosing and refill text message reminders in Diabetes MAP.
Participants self-reported duration of diagnosed diabetes in years and months, and the number and type of diabetes medications prescribed, including insulin. The RA reviewed each participant’s medical record to confirm a T2DM diagnosis and the quantity and type of prescribed medications, and to collect participants’ most recent glycated hemoglobin A1c test result to characterize the sample’s glycemic control.
We assessed participants’ website usage with data logged by Mouseflow ApS™. Specifically, we assessed the total number of days users initiated a session by logging into Diabetes MAP, the total number of minutes users were logged into Diabetes MAP (ie, from the time they created an account until the study was over), and the average number of minutes logged in per days logged in. We also captured whether users set up text message reminders to take their medications or refill prescriptions.
We assessed Diabetes MAP’s usability with 10 items adapted from the Computer System Usability Questionnaire (CSUQ) [
We used SPSS version 21.0 to summarize quantitative data using means and standard deviations (SD), or frequencies and percentages as appropriate. We used selective coding [
The sample (N=32) was on average 51.7 ± 11.8 years of age. Most were female (66%, 21/32), non-Hispanic White (NHW; 60%, 19/32), had at least some college education (88%, 28/32), and were privately insured (78%, 25/32); half had incomes above $50,000 (
Participant characteristics (N=32).
Characteristic |
|
Mean ± SD or n (%) | Range |
Age, y |
|
51.7 ± 11.8 | 26.7-73.4 |
Female |
|
21 (66) |
|
Race/ethnicity | White (non-Hispanic) | 19 (60) |
|
|
Black (non-Hispanic) | 8 (25) |
|
|
Hispanic | 3 (9) |
|
|
Asian | 2 (6) |
|
Education, y |
|
16.3 ± 2.8 | 12.0-24.0 |
Annual household income | Less than $14,999 | 3 (9) |
|
|
$15,000 to $24,999 | 4 (13) |
|
|
$25,000 to $49,999 | 9 (28) |
|
|
$50,000 to $74,999 | 7 (22) |
|
|
$75,000 or more | 9 (28) |
|
Insurance status | Private insurance | 25 (78) |
|
|
TennCare/Medicare | 6 (19) |
|
|
No insurance | 1 (3) |
|
Own a mobile phone |
|
32 (100) |
|
Text message with phone |
|
26 (81.3) |
|
Diabetes duration, y |
|
7.8 ± 6.3 | 0.0-20.0 |
Number of diabetes medications |
|
1.8 ± 0.8 | 1.0-4.0 |
Prescribed insulin |
|
12 (38) |
|
A1c (n=31)a |
|
7.4 ± 2.0 | 4.9-15.8 |
aOne participant did not have an A1c test result in the medical record at the time of data collection.
Among all participants enrolled in the study, the average number of days users logged into the site was 4.2 ± 4.2 days during the 2-week period. The average number of hours logged into the site was 4.3 ± 4.8 hours, and the average time logged in per days logged in was 56.6 ± 47.2 minutes. Five participants (16%) set up text message reminders to take their medications and 4 participants (13%) set up text message reminders to refill their prescriptions.
On the survey, participants rated Diabetes MAP’s usability above average (ie, scores of >3 on a 5-point scale) on each of the 10 items (
Across 9 focus groups, participants shared experiences using Diabetes MAP, including concerns about its usability and recommendations for improvement. Of the 24 unique usability concerns reported, 14 concerns were mentioned in the first focus group and another 6 concerns were mentioned in the second focus group. By the fourth focus group, 95% of all unique usability concerns had been reported. Generally, participants’ experiences with Diabetes MAP were similar across age, gender, race/ethnicity, education, and income.
Survey items assessing Diabetes MAP’s usability, ranked most to least favorably.
|
Related Usability Area | Respondentsa,b, n | Mean ± SD |
The information provided in Diabetes MAP is easy to understand. | Intuitive Design | 28 | 4.3 ± 0.7 |
The information (such as help videos, on-screen messages, etc.) provided in Diabetes MAP is clear. | Intuitive Design | 27 | 4.2 ± 0.8 |
My user interaction(s) with Diabetes MAP are pleasant. | Subjective Satisfaction | 27 | 4.0 ± 0.9 |
Overall, it was easy to learn to use Diabetes MAP. | Ease of Learning | 28 | 3.9 ± 1.0 |
Overall, I feel comfortable using Diabetes MAP. | Subjective Satisfaction | 28 | 3.9 ± 1.0 |
It is easy to find the tools and information that I need. | Intuitive Design | 27 | 3.8 ± 0.9 |
The organization of information in Diabetes MAP is clear. | Intuitive Design | 27 | 3.8 ± 0.9 |
It is easy to navigate the Diabetes MAP website. | Intuitive Design | 29 | 3.6 ± 0.9 |
The information provided in Diabetes MAP is effective in helping me complete tasks on the website. | Efficiency of Use | 27 | 3.6 ± 0.9 |
When I make a mistake in Diabetes MAP, I recover easily and quickly. | Error Frequency and Severity | 26 | 3.5 ± 0.9 |
aNumber of participants providing a response for each item on a scale of 1 (strongly disagree) to 5 (strongly agree).
bSome participants indicated items were “Not Applicable” to their experience.
Participants’ concerns with Diabetes MAP by usability area.
Usability Areas | Concern | Number of Focus Groups Reporting Concern |
Intuitive Design | The site’s layout and content placement was confusing | 4 |
Unnecessary scrolling required to access site features and tasks | 3 | |
Difficult to explore the site using the navigation menu | 3 | |
Unclear how entering information into site tailored the user experience | 1 | |
Unclear how to minimize navigational videos | 1 | |
Location of navigational videos was confusing | 1 | |
Ease of learning | Instructions for accessing and using the site were unclear | 4 |
Directions for use within the site were unclear | 3 | |
Navigational videos did not help with accessing features/completing tasks | 3 | |
Navigational videos dysfunctional | 1 | |
Efficiency of Use | Unable to save progress with completing tasks | 3 |
Website pages took a long time to load | 3 | |
Difficult to select time-zone using worldwide map | 2 | |
Automatically logged out of site if stopping use for 20 minutes | 2 | |
Website not compatible with other digital devices (eg, iPads®) | 2 | |
Difficult to scroll through different site windows | 1 | |
Error Frequency and Severity | Website not compatible with different browsers | 7 |
Error messages encountered while trying to log in | 6 | |
Difficult to search for medication names in medication list | 3 | |
Difficult to search for medication doses in medication list | 3 | |
Technical support was required to use website | 1 | |
Memorability | The site’s purpose was unclear | 4 |
Website URL was confusing and made accessing the site difficult | 3 | |
Subjective Satisfaction | The site had a non-user-friendly interface | 2 |
An intuitively designed website is easy to navigate and understand [
I didn’t even realize there was a video connected to [the image] until you pointed out that arrowhead.
In another instance, participants were unclear how information they entered into the site affected their user experience. For example, data entered at account creation (eg, entering one’s time zone and mobile phone number) impacts functionality elsewhere on the site (eg, receiving text message medication reminders in the appropriate time zone), and data entered into the IMB model-based barriers-to-adherence assessment impacts what videos and content are available for viewing in a user’s My Tailored Tools section of the site. As a result, some participants did not access or use certain parts of the site.
It was also common for participants to miss out on site features and functions entirely (eg, the option to print one’s medication list or set up text message reminders) because they were unable to locate them.
I am really frustrated because I would have loved [text message reminders]. I’m serious. Where was it?
Related to this issue were concerns with navigating between different types of content in Diabetes MAP. It was common for participants to describe difficulty reading task instructions, viewing educational videos, and using features on a single webpage. In 1 focus group, participants commiserated with one participant who said she could have used more assistance with exploring the site:
I [would have liked] more instructions to help navigate [the medication list] and clearly access the site.
Unintuitive design issues such as this one made it difficult for participants to successfully use and fully engage with Diabetes MAP.
Ease of learning refers to how fast new users of a website can learn and accomplish basic site tasks [
[I] wasted a lot of time…. It had dragged on for 2 or 3 days when I could have actually been using [the site] and I had to contact you, which I didn’t really want to have to do. It was frustrating, to say the least, and I just felt like, “What’s wrong with me? What’s wrong with my computer?”
Other participants voiced confusion without frustration such as this participant who said the directions to enter one’s medications were confusing.
It wasn’t a huge challenge, but in the medication list, it didn’t specify if it wanted you to put in just your diabetes medicine or other medicines, so I put in all my medicines.... It would have been nice if it was more specific.
In the most extreme cases, some participants said the navigational videos did not help them, particularly when videos did not work. When asked about these videos, members of 1 focus group were united in their unsuccessful experience.
I never could get [the video] to play.
And I couldn’t either, and…I thought maybe it was my computer, but it wasn’t, it was [the videos] I guess.
Efficiency of use refers to how quickly a user can complete website tasks [
I felt like it was a little heavy to start with… iPads® can open it, but [it] needs a lot of time, even though I have high speed [Internet]. When you open it [on the] iPad®, you can’t get some clips unless you are [using] a desktop or laptop.
Other participants speculated loading delays were due to the size and volume of videos being streamed.
[The navigational and educational] videos take a lot of feed. It takes forever to load, and when you click [one], it doesn’t immediately work.
In some instances, participants were unable to save their progress on a task to revisit it and complete it later. The site also logs users out who are logged in, but who do not use the site for 20 minutes, which resulted in several participants losing task progress for partial completion.
If you do half of it, and you try to do something else, and the computer freezes or logs you out, you have to start all over again. Is there a possible way—I’m sure there is—to save it and come back to it to finish it?
Finally, some participants felt certain tasks were overly complicated and time-consuming. For example, the site asks users to select their time zone on a worldwide map instead of from a more efficient drop-down menu.
When it got to the time zone, and that little map came up…I was thinking if they had a drop down for [it]—it would probably be easier—instead of a map.
Error frequency and severity refers to how often users make errors, the seriousness of the errors, and how users recover from errors [
It was really weird. I have multiple browsers of Internet Explorer®, and I kept trying to change them, thinking maybe I’m just not using the right compatibility thing. Finally, I was like, “OK, I’m just not going to look at this [website].”
Errors while creating and logging in to user accounts were mentioned in 6 out of 9 focus groups. To create an account, users are required to enter personal information using several entry methods including text fields and drop-down menus. If information is entered incorrectly, users receive error messages preventing further access. The recovery time needed to overcome these errors varied between participants. When participants were unable to access Diabetes MAP, some enlisted professional and nonprofessional technical support.
I just happened to know this computer guy who was coming in my [office] to do some other work and I asked him … I said, “Can you get this website up?” It took him a while, and this is all this man does is IT work.
Other participants reached out to study personnel who answered questions and provided remote assistance consistent with the written instructions participants were provided on how to access the site. Participants who encountered errors and did not seek assistance reported frustration and wasted time, causing some to give up using the site altogether.
A website with memorability is one users can remember well enough to use it effectively in the future [
I searched for Diabetes MAP on the medical center’s site and got directions for how to get to the Diabetes Center.
Despite being both told about the website’s intent and receiving an instructional handout with this information, participants in nearly half of the focus group sessions felt the website’s purpose was confusing. In some cases participants forgot Diabetes MAP’s purpose altogether, which led to using the website in unintended and ineffective ways.
I didn’t even realize it was just for taking medications until we came to this focus group.
Subjective satisfaction is determined by how much the user enjoys using the website [
I’m not an IT person, but I’m a supervisor in my department, and I do not have a problem [with computers].... But this one over here, it was like going against a brick wall, OK? It was not user-friendly, whatsoever.
It is important to note that subjective satisfaction concerns were limited and mentioned in only 2 of the 9 focus groups. Participants who successfully accessed the site’s features and tasks enjoyed it. Participants across focus groups highlighted several positive aspects of the website:
[Diabetes MAP] reinforced some of the things I knew, but also gave me some new information, so I thought that was very good—I really enjoyed the educational features. I thought they were very helpful, [such as] what [medications do] to your body [and] how to take your medications.
I liked the skills [section], where it showed those 4 people and their tips on when to take the medicine.
I liked the My [Tailored] Tools part—I liked being able to read about you know, the consequences of not taking care of yourself when you have diabetes…they had some tips that I found…helpful. I love the text message notifications—that has increased my compliance.
When participants voiced concerns about Diabetes MAP, they also gave suggestions for improving it. Common across all suggestions was a request for more simplicity and flexibility within the site. To improve the site’s ease of learning and efficiency of use, participants wanted more straightforward methods for accessing the site’s components. They suggested strategically placing instructions, features, and Web content in easily recognizable ways. Some participants wanted specific features to be accessible on every page, or clearly designated on their own page. For example, participants liked the idea of a page dedicated to setting up text message reminders. On this page, previously entered medication information (ie, name, dosage) would appear and users could set up reminders for when to take specific medications and order prescription refills.
To improve the site’s intuitive design, participants recommended the navigational videos be minimized or eliminated entirely.
[The navigational video] is a distraction for me because if I have to scroll down for whatever I have to do, it would be better for me if the video came up once you click it, and is then minimized.
Additionally, in reference to the site’s memorability, participants recommended using a simple and recognizable URL that is easy to locate with an online search. They also wanted Diabetes MAP’s purpose to be clear while using it (eg, spelling out the Diabetes MAP acronym and including images of diabetes medications throughout the website).
Participants wanted a simplified, streamlined Diabetes MAP user experience. In order to improve the site’s error frequency and severity, participants recommended increasing compatibility across multiple browsers, including older versions of commonly used Web browsers. Additionally, they requested the ability to use Diabetes MAP across multiple digital devices without loading time delays. They also stressed the importance of clear and accessible resources for user support (eg, the ability to contact study staff directly if they had issues or questions, an accessible and searchable help resource on the website itself). Finally, participants wanted functional, useful, and easy to recall navigational videos to further facilitate learning how to use the site.
Those who reported some dissatisfaction, but generally endorsed the utility of Diabetes MAP, felt it might be more appropriate for certain types of patients with diabetes. For example, some users diagnosed with diabetes for a longer period of time felt the website might be particularly helpful for newly diagnosed patients. Other users suggested the website might be more useful for younger, more technology-proficient patients who prefer technology-delivered information as opposed to more traditional print materials.
Usability testing is issue-focused and designed to assess the extent to which users can easily, efficiently, and effectively perform tasks with a technical system. We employed a mixed-methods approach to understand the challenges of using a Web-delivered medication adherence promotion intervention called Diabetes MAP. Participants with diabetes provided ratings and descriptions of their experiences using the website, as well as recommendations for improving it. On surveys, participants agreed Diabetes MAP was helpful and easy to use, but, in focus groups, they mentioned 24 unique user concerns related to each of the 6 factors determining website usability [
Our quantitative results are comparable to other usability studies employing the CSUQ, in that total ratings were above average [
Recent usability studies of Web-delivered interventions for T2DM self- management yield results comparable to ours. In their evaluation of a Web-based dietary intervention, Ramadas et al found positive ratings of a website’s usability based on survey items; however, this study did not use qualitative assessments [
Our research highlights the value of using mixed methods for usability testing. Had we relied on only survey data, we would have incomplete information on Diabetes MAP’s usability. Collecting qualitative data as part of usability testing reveals insights on unanticipated challenges and ideas for improving a site [
The US Department of Health and Human Services (HHS) has set forth peer-reviewed guidelines for improving the design and usability of websites [
Our second principle is to design websites with the goal of keeping users informed and aware of website processes. As a general observation from our focus groups, users became frustrated with unanticipated incidents (eg, long downloading times, automatically being logged out). In some cases, it may not be possible to reduce the size of a page to minimize the time it takes for a webpage to load [
There are limitations to our study. Because we recruited our sample from a single academic medical center, our findings may not generalize to other patient populations. However, our sample characteristics map onto the academic medical center’s patient population for whom Diabetes MAP was designed for. Additionally, although we were able to track time spent using Diabetes MAP, we were unable to track how participants used their Vanderbilt patient portal account because the two websites are not integrated. Considering the recent advancement of patient-provider communication in Web-delivered interventions, it will be valuable to track usage with this type of feature in future studies
Our findings highlight the importance of evaluating a website’s usability prior to testing its efficacy. For Web-delivered interventions to be used as intended, researchers, Web designers, and developers must plan sufficient time to perform usability testing [
hemoglobin A1c
Computer System Usability Questionnaire
Diabetes Medication Adherence Promotion
Information-Motivation-Behavioral skills
glycated hemoglobin A1c
US Department of Health and Human Services
research assistant
Research Data Capture
standard deviation
type 2 diabetes mellitus
non-Hispanic White
This research was funded with support from the Center for Diabetes Translational Research (NIDDK P30 DK092986) and CYO’s Career Development Award (NIDDK K01 DK087894). CYO and LAN are both supported by NIDDK R01 DK100694.
Author LAN wrote and edited the manuscript. Author MCB co-designed the study, facilitated focus group discussions, collected, coded, and managed data, and conducted analyses. Author AL co-wrote and edited the manuscript. CYO co-designed the study, oversaw data collection and analyses, and co-wrote and edited the manuscript. The corresponding author and guarantor of the manuscript (CYO) takes full responsibility for the work as a whole, including the study design, access to data, and the decision to submit and publish the manuscript.
None declared.