The analytic work that provided the foundation for the first interactive prototype entailed the review of ADA guidelines [11] to determine priorities for the selection of patient facts that would need to be pulled into the T2DM app from the EHR. The team also conducted interviews with 8 conveniently selected clinical leaders from care delivery organizations of significant scale to learn about the intended users, the context of use, and the specific workflow in their EHR associated with reviewing patient facts and ordering medications to manage T2DM.

Once the initial prototype was developed on the basis of findings from the analytic phase, 8 iterative rounds of user testing were carried out to get feedback on different parts of its evolving design. These were conducted remotely from May 4 to October 2, 2020, owing to COVID-19 restrictions, and each session was recorded. For each round of user feedback, a sample of prescribers were included in 1-hour, one-on-one test sessions with the prototype app. In total, 16 participants were recruited directly from a single large provider network, and the remaining participants were recruited via a national third-party recruitment service. The latter were compensated at the prevailing market rate for physicians, physician assistants, and nurse practitioners. In total, 25 MD physicians, 11 nurse practitioners, and 15 physician assistants participated in the user testing. We used a talk-aloud method for data collection while prescribers reviewed and ordered medications with the prototype.

In addition, during 5 of the 8 rounds of user testing, participants completed the System Usability Scale (SUS) questionnaire at the end of their sessions (for logistical reasons [sample size and participant time], the SUS was not administered in rounds 5, 7, and 8). The SUS questionnaire is an industry standard for evaluating the usability of software applications consisting of 10 statements with 5 response options (ranging from “Strongly Disagree” to “Strongly Agree”) to each question. The statements are as follows:

The SUS survey yields a single number that represents a composite measure of the overall perceived usability of the system. SUS scores have a range of 0 to 100 and the score is a relative benchmark that is used against other iterations of the system. The SUS is a reliable and valid measure of system satisfaction. Sauro [12] reports that the average SUS score from 500 studies across various products (eg, websites, cellphones, and enterprise systems) and across different industries is 68. A SUS score above 68 is considered above average and anything below 68 is below average.

In total, 43 participants completed the 2-hour remote evaluation study of the prototype T2DM app that was developed and refined through the user testing phase. Participants were recruited via a national third-party recruitment service. The study population included 21 MD and 22 non-MD physicians (5 nurse practitioners, 5 physician assistants, 6 nurses, and 6 pharmacists). To be included in the study, candidates were required to (1) have at least 1 year of experience treating T2DM, preferably in a family or internal medicine practice; (2) currently prescribe, prescribe on behalf of, or provide medication recommendations as part of their current role; and (3) currently interact with 15 or more patients per day. Participant compensation was set at the prevailing market rate for physician assistants, nurse practitioners, pharmacists, and nurses.

Medical and legal review were conducted to ensure no aspect of clinical or legal regulations or ethical considerations were overlooked. Formal institutional review board or ethical review was not required in the study because no protected health information was included, and participation was limited to usability testing and providing feedback about the app. No private information about participants was collected or included in the analysis and study. All participants gave verbal consent and were reimbursed for their time.

A Universal Design framework was followed for accessibility and to accommodate a wide range of people (eg, people with color blindness), and care was taken to convert typical in-person interaction to virtual to avoid exposure to COVID-19.

The prototype T2DM app (Figure 1) pulls data from the EHR to present patient information in a user-friendly way on the left side of the screen. The right side of the screen displays a list of current diabetes medications, evidence-based recommendations based on the latest ADA guidelines, and a table of on- and off-guideline medications for ordering. In addition to the prototype T2DM app, a second prototype that is a close representation of a commercial EHR user interface was developed for use in the validation study (Figure 2). Both prototypes (the mock EHR and the T2DM) presented mock patient facts that simulate patients with T2DM. The workflow was captured as each participant interacted with both prototypes (Figures 3 and 4).

The study was conducted between March 15 and 19, 2021, and because of continued COVID-19 social distancing restrictions, the test sessions were conducted remotely via a usability testing platform (Loop11) or via a web conference tool (Zoom or WebEx). One independent moderator was present with the participant throughout the session. Several sessions were observed by members of the product development team; two of the observing team members and the moderator asked participants clarifying questions during the session. The participants employed their own computers to display and interact with the apps. Although the environment with respect to seating, lighting, sound levels, temperature, and humidity varied, participants were most generally seated at a table or desk in a personal space within their home. In most cases, the participants shared their screens and interacted with the apps on their own desktops. In several cases, the moderator shared their screen and passed control of mouse and keyboard to the participant.

The method of data capture depended on the platform employed in the session. The Loop11 platform captured videos, task time, and screen review data; the web conference tools captured video only. In all cases, a notetaker recorded participant responses, and medication orders were collected manually from screen shots taken during the time of the study and determined by video review.

Participants also completed the Kano Model Survey. The Kano Model is a tool that can be used to prioritize the critical to quality characteristics, as defined by the voice of the customer [13]. The three categories identified by the model are as follows: (1) Must-Have: whatever the quality characteristic is, it must be present, such that if it is not, the customer will go elsewhere! (2) Linear or Performance: the better we are at meeting these needs, the happier the customer is. (3) Exciter or Delighter: those qualities that the customer was not expecting but received as a bonus.

The task data collected included “pass” or “fail” (medication order adheres or does not adhere to evidence-based recommendations), task time, subjective comments from the Kano Model Survey, and video recordings of the computer screen and audio.

The study employed a 2 (Group) × 2 (Time Period) mixed model simulated-use design (shown in Figure 5). Participants were randomly assigned to one of two groups (Traditional, Tool) with the constraint that each group was comprised of 50% MDs and 50% other medical licenses. Participants in both groups were asked to review two sets of eight mock patients and order diabetes medications. Participants in the Traditional group employed the mock EHR for reviewing and ordering in Period 1 and employed the prototype T2DM app for reviewing and ordering in Period 2. Participants in the Tool group employed the prototype T2DM app in both time periods. Both groups were afforded the opportunity to access any online resources that are typically used as part of their medication decision-making (eg, websites such as UpToDate, ADA evidence-based guidelines).

Each reviewing and ordering period was followed by administration of the NASA TLX to measure the perceived cognitive load associated with that performance period [14]. The NASA-TLX solicits ratings for mental demand, physical demand, temporal demand, performance, effort, and frustration on a scale from “Very Low” to “Very High.” The web-based interface presented the scale as 10 radio buttons that were not numbered; responses were coded 0-100 consistent with extant literature [14]. Pairwise rankings of the indices were collected to generate weights for computation of the TLX score.

The data were analyzed to test the hypothesis that providers would more frequently align their orders with the ADA evidence-based guidelines when using the T2DM app, and that the process would be more efficient and less taxing. The hypothesis predicts performance advantages for the Tool group during the first period, and a greater change in duration for Traditional group in Period 2.

Each 2-hour session was conducted individually and began with guidance about the resolution of technical difficulties associated with the employed platforms and the informed consent process. The overall study process for both participant groups is shown in Figure 3.

We analyzed data using SPSS (version 25.0; IBM Corp). Dependent variables in the study (percent adherence to evidence-based guidelines, task time, screens reviewed, and perceived workload rating) were all subjected to a 2 × 2 mixed model analysis of variance (ANOVA) with Period (Period 1 vs Period 2) as the within-subjects factor and Group (Traditional vs Tool) as the between-subjects factor. Statistical significance was accepted at a level of P<.05.

Each round of user testing focused on different elements of the prototype app while also iteratively reviewing changes that resulted from the previous round or rounds. Changes became smaller and more focused with each successive round of testing until the final version was reached after round 8. Table 1 provides information about each round.

Comments made by user testing participants (Table 2) were used in determining design changes for each round.

The SUS scores were “average” or “above average” on each of the rounds of user testing where measured (Table 3). While the averages for rounds 1-4 were all above 68, the lower average for round 6 resulted from using an alternate design that participants perceived as less user-friendly; this design was subsequently abandoned in favor of the earlier design.

In total, 14 of the validation study participants responded to the Kano Model Survey (Figure 6).

Orders were scored as pass if the participant ordered any medications that were on-guideline, discontinued any medications not aligned with guidelines (ie, a recommended discontinuation) or if the guidelines recommended against medication changes, and fail if any ordered medication was off-guideline.

The mean percentage adherence to ADA evidence-based guidelines is shown in Figure 7. There was a main effect of Group on adherence to guidelines (F_1,41=8.99, P=.005, =0.18). As hypothesized, medication ordering by the Tool group was more frequently aligned with guidelines (mean 92.1) than it was by the Traditional group (mean 84.2). Furthermore, there was a main effect of Period on adherence to guidelines (F_1,41=37.63, P<.001, =0.48). Medication ordering in Period 2 was more frequently aligned with guidelines (mean 93.7) than it was during Period 1 (mean 82.6).

However, there was a significant Group × Period interaction (F_1,41=45.43, P<.001, =0.53). For the Traditional group, ordering was more aligned with guidelines during Period 2, when the T2DM Tool was first introduced (mean 96.2), than it was in Period 1 (mean 72.0; t₂₀=–8.47, P<.001). For the Tool group, ordering was similarly aligned to guidelines during Period 1 (mean 92.6) and Period 2 (mean 91.5; t₂₁=0.46, P=.65).

Of particular interest was a planned comparison between groups during Period 1. As hypothesized, medication ordering by the group using the prototype T2DM app was more frequently aligned to ADA evidence–based guidelines (mean 92.6) than medication ordering was by the group using the mock EHR (mean 72.0; t₄₁=–6.20, P<.001).

Measurement of task times were impacted by technology issues (ie, poor internet connections), which resulted in extreme task times. Extreme task times were excluded based on a cutoff of 3 SD. Task time SDs were calculated using all available task time data. Task times greater than (and less than 3 SD) were excluded from the data analysis (ie, from the computation of each participant’s mean for each period). The percentage of trials removed by the 3 SD trim was 1.8%. There were no task times 3 SD below the mean because that was a negative value.

The mean task time used for the completion of orders is shown in Figure 8. There was no main effect of Group on task time (F_1,41=1.46, P=.23, =0.03), but there was a main effect of Period (F_1,41=26.70, P<.001, =0.39). Medication ordering time was lower during Period 2 (mean 117.2) than it was in Period 1 (mean 133.4).

There was also a significant Group × Period interaction (F_1,41=5.23, P=.03, =0.11). The Tool group completed prescription orders more quickly during the second period (mean 107.7) than the first period (mean 130.7; t₂₁=6.49, P<.001). For the Traditional group, task time did not significantly differ between Periods 1 (mean 136.1) and Period 2 (mean 127.2; t₂₀=1.74, P=.10). Finally, Period 1 task times also did not significantly differ between the two groups (t₄₁=0.50, P=.62).

For technical reasons, screen review data was not captured for 9 participants from the Traditional group and 5 participants from the Tool group.

There was no main effect of Group on the number of screens reviewed (F_1,26=0.51, P=.48, =0.02), and there was no main effect of Period (F_1,26=0.73, P=.40, =0.03). Because the variance was found to be unequal across groups (by Box M and Levene tests), to reduce the probability of type 1 error, the α level was adjusted accordingly to be increasingly conservative (α’=.001). Based on the adjusted α level, there was no Group × Period interaction effect (F_1,26=7.16, P=.01, =0.22).

Of the 43 participants, 9 participants from the Traditional group and 6 participants from the Tool group were not able to complete the survey for logistical or technical reasons.

There was no main effect of Group on the NASA TLX score (F_1,34=2.30, P=.14, =0.06), and there was no main effect of Period (F_1,34=2.13, P=.15, =0.06). There was no significant Group × Period interaction (F_1,34=0.23, P=.64, =0.01). Finally, the perceived workload associated with Period 1 did not significantly differ between the two groups (t₃₄=1.14, P=.26).

As part of our analysis, we attempted to evaluate the effect of provider type on the studies using mixed ANOVAs with the clinical role (MD vs non-MD physicians) between subjects and test period (Period 1 and Period 2) within subjects. There was no significant main effect of clinical role on adherence to guidelines (F_1,41=4.05, P=.05, =0.09), and role × time interaction (F_1,41=1.87, P=.18, =0.04). Furthermore, there was no significant main effect of clinical role on test period (F_1,41=3.86, P=.06, =0.09), and role × time interaction (F_1,41=.79, P=.38, =0.02). Provider type analysis was not conducted for other parts of the study, partly owing to the sample size and initial findings.