The traumatic brain injury–CDSS delivers a patient-centered outcomes research guideline for venous thromboembolism prevention in patients with traumatic brain injury [2,25], which was centered around the modified Berne-Norwood criteria [8]. Patients are categorized as being at low, moderate, or high risk for spontaneous progression of hemorrhage, with prophylactic treatment regimens tailored for each group, respectively. The traumatic brain injury–CDSS is a modification of an original, natively developed COVID-19 CDSS logic model, which is currently translated to interoperable standards [16] (Figure 1). We chose this model for two major reasons: (1) we have previous experience implementing the traumatic brain injury venous thromboembolism prevention guideline with success [2], and (2) many of the data elements and artifacts (~80%) overlap with minimal additional mappings and technology build. The final CPG-FHIR (clinical practice guidelines on Fast Healthcare Interoperability Resources) COVID-19 CDSS included 108 data elements. The traumatic brain injury clinical practice guideline added 21 data elements specifically related to traumatic brain injury (eg, intracranial pressure monitoring, neurosurgical consultation, craniotomy, head computed tomography imaging, subdural hematoma, contusion, and Glasgow Coma Scale).

Given the logic model, patients will enter the system by admission under predetermined diagnoses, generally falling under the traumatic brain injury category. Our target population is adult patients admitted with an acute traumatic brain injury, as defined by ICD-10-CM (International Statistical Classification of Diseases, Tenth Revision, Clinical Modification): S06.1-S06.9 or S06.A. Patients who died within 24 hours of hospital admission and patients documented as “comfort cares” during the first 72 hours of hospitalization will be excluded. Additionally, patients with a pre-existing venous thromboembolism or inferior vena cava filter at the time of admission and patients with a mechanical heart valve or ventricular assist device will be excluded from the final analysis.

The CDSS is provided in the subsequent 72 hours (dependent on risk stratification) and administered in the form of seven alerts (Textbox 1). Textbox 1 provides descriptions of the seven alerts deployed at various points in time, stratified by level of risk.

Our foundational usability evaluation-related research work was conducted by a team of researchers from the University of Minnesota (UMN) and associated health systems, such as the M (Minnesota) Health Fairview and affiliated hospitals, which consisted of 9 state-verified trauma centers, including the flagship UMN as an American College of Surgeons–verified level 2 trauma center. The usability evaluation methodology is reported in Figure 2.

Our team of experts was comprised of builders or IT specialists (SB, PM, and MS) and clinicians (SS, RFR, and CJT), who helped with building the 3 master test patients stratified by risk level (Textbox 2). Scenarios were generated by a team of clinicians having diverse sets of experiences (ie, CJT, a trauma surgeon, and project principal investigator, SS, a hospitalist, and RFR, a physician informaticist with user acceptance testing experiences) and builders or IT specialists (SB, PM, and MS) after in-depth discussions. The goal was to have scenarios that are commonly encountered and require trauma providers to perform certain high-priority tasks. Copies of the master patients were created to be used by each study participant..

The purpose of the testing was to receive user feedback effectively and efficiently, ensuring that the testing context matches as much as possible with the user interactions in the real world. Leveraging the Epic test environment to create a simulated environment came with a few constraints described under limitations.

This component of the project is not a clinical trial. This study protocol (STUDY00017107) was submitted to the UMN Institutional Review Board and given the determination of “exempt” as secondary research for which consent is not required. The mixed methods investigation was determined as a “not human research” as a quality improvement activity. Signed and written informed consent was obtained to screen and audio record the session, with the speech-to-text transcription feature turned on. Each recording was saved on a secured and encrypted UMN device. Each participant later received a US $50 gift card for their participation.

Five trauma providers participated in the individual usability testing session. The detailed characteristics are described in Table 1.

We identified a set of both common and unique issues using 2 methods: heuristic evaluation with experts (domain experts, ie, physicians, usability experts, or both) and usability testing with end users (the actual users).

Thirty-seven heuristic violations (23 unique and 14 duplicates) were identified by 3 experts. The results showed that 30% of possible heuristics were responsible for ~74% of all violations, the Pareto principle (Figure 3). Three heuristics that represented the most violations were (1) consistency and standards (n=8, 35%), (2) help and documentation (n=5, 22%), and (3) recognition rather than recall (n=4, 17%). Of 23 unique, we found cosmetic (n=6) and minor (n=17) issues only.

We identified 108 issues (49 unique and 59 duplicates). Of 49 issues, we found cosmetic (n=6), minor (n=20), major (n=10), and catastrophic (n=2) issues. The remaining 11 were considered operational and not ranked. The mean responses to SEQ for all tasks were 5.9 (SD 0.53), ranked from 1 (very difficult) to 7 (very easy). The mean SUS score was 77.5 (SD 16) interpreted as an acceptable or good usability range (Figure 4). The 95% CI for the mean SUS score is 77.5 (95% CI 57.6‐97.4).

Between the 2 methods, we identified a total of 145 issues, 66 of which were unique: 17 from heuristic evaluations, 43 from end users, and 6 common issues shared between both methods. Thematic analysis was performed on all unique issues, eliminating those arising from the constraints of thfsuye Epic test environment (66‐11=55).

We ranked 55 unique usability issues on a severity scale from 0‐4, where 20% (n=11) were marked as cosmetic; 58% (n=32) were identified as minor; 18% (n=10) as major, and 4% (n=2) as catastrophic issues. We identified both cosmetic and minor issues through heuristic evaluations and usability testing with end users. However, severe and catastrophic issues were solely identified during the usability sessions with end users.

Thematic analysis was conducted on 66 unique issues identified through heuristic evaluations and usability testing. Eleven issues arising from Epic test environment constraints were eliminated. We identified 13 unique themes, including lack of supporting evidence, operational barriers arising from the test environment, formatting inconsistencies and lack of following standards, suboptimal data retrieval and display, lack of user control, lacking language clarity, order set-related suboptimal usability, language inconsistencies, best practice alert lacking patient-specific details, clinicians missing critical information or action, unfamiliarity with the full clinical context, infinite ordering loop leading to error, and having redundant options. The 3 most identified themes were lack of supporting evidence (n=17, 26%), operational barriers arising from the test environment (n=11, 17%), formatting inconsistencies, and lack of following standards (n=8, 12 %; Table 2).

Lack of supporting evidence refers to the end users’ request to integrate and access evidence-based citations throughout the CDSS. Of the 17 usability issues about evidence-based medicine, 14 were identified by end users and 3 were identified by experts. Examples include users disagreeing with threshold values and requesting evidence-based medicine citations, and users requesting the ability of the CDSS tool to be explained.

Formatting consistency and a lack of following standards refer to the system’s ability to present information clearly and concisely, with a consistent interface to ensure easy readability for the user. Of the 8 usability issues about formatting consistency and standards, 2 were identified by end users, and 6 were identified by experts. Examples include inconsistent and difficult-to-read text usage, style, color, and sizes.

Suboptimal data retrieval and display refer to the system’s ability to present information in a manner that reduces user cognitive burden by increasing smooth and efficient navigation by auto-populating data (such as laboratory results). Of the 6 usability issues about easy retrieval and optimal display, all were identified by end users. Examples include users requesting that laboratory or imaging values and administration time be incorporated into the CDSS.

Lack of user control refers to the flexible design of the system to ensure smooth interaction with the user. Of the 5 usability issues about user control, 2 were identified by end users and 3 were identified by experts. Examples include users requesting the ability to navigate away from the tool and forgiveness when using the CDSS.

Lacking language clarity refers to specifying verbiage to omit any confusion the user may have. Of the 4 usability issues about language clarity, 3 were identified by end users and 1 was identified by experts. Examples include specifying verbiage used throughout the tool and using specific phrases (ie, the use of “active bleed” vs “new bleed” may imply 2 different ideas).

Order set-related suboptimal usability refers to any suboptimal interactions the user may encounter when interacting with the order set. Of the 4 usability issues related to order set-related suboptimal usability, all were identified by end users. Examples include users requesting the tool display prompts regarding its functionality on a case-by-case basis and easy access to the admission order set.

Language inconsistencies refer to the consistent use of verbiage throughout the system. Of the 3 usability issues about language consistency, they were all identified by experts. Examples include discrepancies between verbiage used in the storyboard or title and the consistent use of terms implying the same definition (ie, either “snooze” or “lockout time” should be used, not both).

Best Practice Advisory (BPA) lacking patient-specific details refers to information that the system does not include, which users would prefer to be included. Of the 2 usability issues about BPA lacking patient-specific details, both were identified by end users. Examples include adding patient risk stratification to the BPA (ie, changing the tool to be more diagnostic) and including crucial patient-specific laboratory values.

Clinicians missing critical information refers to ensuring a system design that ensures easy readability for the clinician. Of the 2 usability issues about eliminating the chance of clinicians missing critical information, 1 was identified by end users and 1 was identified by experts. Examples include bolding information and making information more accessible to improve readability.

Unfamiliarity with the context refers to additional guidance presented to the clinician on the system. Of the 2 usability issues about unfamiliarity with context, both were identified by end users. Examples include ensuring proper education and introduction of the tool to clinicians to ensure it does not interrupt their workflow.

A redundant option refers to options already included as part of the system. Of the 1 usability issue about redundant options, it was identified by end users. Examples include omitting the “already ordered complete” button.

An infinite ordering loop leading to error refers to an infinite loop that occurs when using the system, during the beginning of its launch. It has since been resolved as of May 16, 2023. The 1 usability issue about the infinite ordering loop was identified by end users. Examples include users noting their warranted frustration with the ordering process.

Clinicians most often identified issues relating to the readily apparent integration of evidence-based medicine into the CDSS, recognizing a need for the use of citations and principles. Clinicians frequently noted how the CDSS might disrupt their workflow. For practical feasibility, request the integration of critical laboratory or imaging values and consistency in formatting throughout the tool to reduce user cognitive burden and ensure an efficient workflow. A compilation of unique usability issues categorized by theme is included in Multimedia Appendix 1.

Figures 5 and 6 present a few examples of modifications made based on insights from heuristic evaluations and usability testing by end users. The full list of identified usability issues is provided as a supplementary table.

Using a dual usability evaluation approach, namely expert-driven heuristic evaluation and end user–driven usability testing, we identified a variety of both similar and unique issues across the traumatic brain injury venous thromboembolism CDSS. Of the 66 unique issues identified, the percentage attributed to each method was as follows: usability testing (43 issues, 65%), heuristic evaluations (17 issues, 26%), and both methods together (6 issues, 9%). Although the mean severity scores of the identified problems did not differ significantly between the 2 methods, issues that were ranked as severe and catastrophic were exclusively found through usability testing by the end users. The most common issues identified across both methods were related to a lack of supporting evidence, operational barriers, and formatting consistency or lack of following standards. Heuristic evaluations primarily identified issues related to formatting consistency or lack of following standards, user control, suboptimal language clarity, and consistency. In contrast, usability testing revealed a broader range of issues, where lack of supporting evidence, operational barriers, and suboptimal data retrieval and display were found to be the most common.

Using a combination of these 2 methods can identify a wider range of usability issues. This aligns with prior evidence suggesting that combining 2 methods of usability evaluation can facilitate the development of an optimal CDSS [24]. We found that the majority of issues identified by usability testing were only discovered by end users, as compared to the experts. Although the experts identified fewer issues, they pinpointed more granular issues related to the layout, formatting, and consistency. In contrast, end users were more concerned with larger, more consequential issues, such as the lack of supporting evidence to increase their trust and confidence and fragmented workflow (primarily sending form test environment-related constraints). The findings from this study validate many of the claims made in previous comparisons of heuristic evaluations and usability testing [31,32] such as heuristic evaluations being more cost-effective in terms of time and resources, and the ability of each method to identify distinct sets of usability issues with very small convergence rates (9% in our study). Heuristic evaluation findings were limited to identifying issues, whereas end users engaged in usability testing provided both strengths and weaknesses. For example, end users appreciated the clarity of the premier questionnaire (which covered the critical issue for risk profiling), as well as the straightforward alert verbiage. They also appreciated that the creatinine clearance value was apparent in the laboratory order results, eliminating the need to navigate to find it.

There are some limitations to this study. First, the usability evaluation was conducted at a single institution (M Health Fairview) using a single electronic health system (Epic). We had 5 clinicians participate in the end user testing. Several usability evaluation resources suggest that testing with 5 individuals is often considered appropriate for identifying usability issues, especially in qualitative analysis [26]. However, a larger sample size would have strengthened the generalizability of the findings for quantitative analysis. Additionally, we did not examine whether participants’ amount of experience in their respective fields or their overall familiarity with digital technology influenced their insights.

Usability testing being conducted in the Epic test environment introduced some constraints that users had to adjust to, such as manually shortened refresh time frames to decrease testing time, which resulted in some unusual system behaviors. To simulate the passage of time, users had to right-click anywhere and hit “refresh” after responding to alerts. For the testing session, test results and imaging reports were simplified and appeared shorter in length. Consequently, the insights gathered during the simulation may not completely represent the realities and challenges faced by clinicians working in high-pressure environments, caring for their patients who are critically ill. Finally, the IT team faced challenges due to system constraints, as the systems were hardwired with limited ability for upgrades. We were, however, able to incorporate several build-related upgrades through close collaborations. We are still in the process of upgrading the tool based on the observations made during our evaluations and the feedback collected from end users since the tool went live.

Concerning further CDSS implementation in different hospital settings, we plan to scale and assess the effectiveness of the CDSS for venous thromboembolism prophylaxis guidelines in patients with traumatic brain injury. This will consist of a stepped wedge hybrid effectiveness implementation trial to scale the CDSS across 9 sites within 4 trauma systems.

By using dual usability evaluation methods, leveraging both expert and end user input during the early phases of system development in an iterative manner, we were able to identify a diverse range of usability issues. These issues varied in severity and were relevant to each evaluator based on their respective experience and role. Through conducting this study and our continued efforts to gather feedback from end users, we have been able to optimize the design and functionality of the traumatic brain injury–CDSS to align more effectively with clinicians’ needs and workflow. This comprehensive approach has provided us with valuable insights in refining CDSS, ultimately improving its usability and effectiveness in clinical practice.