This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on http://humanfactors.jmir.org, as well as this copyright and license information must be included.
As the electronic health record (EHR) becomes the preferred documentation tool across medical practices, health care organizations are pushing for clinical decision support systems (CDSS) to help bring clinical decision support (CDS) tools to the forefront of patient-physician interactions. A CDSS is integrated into the EHR and allows physicians to easily utilize CDS tools. However, often CDSS are integrated into the EHR without an initial phase of usability testing, resulting in poor adoption rates. Usability testing is important because it evaluates a CDSS by testing it on actual users. This paper outlines the usability phase of a study, which will test the impact of integration of the Wells CDSS for pulmonary embolism (PE) diagnosis into a large urban emergency department, where workflow is often chaotic and high stakes decisions are frequently made. We hypothesize that conducting usability testing prior to integration of the Wells score into an emergency room EHR will result in increased adoption rates by physicians.
The objective of the study was to conduct usability testing for the integration of the Wells clinical prediction rule into a tertiary care center’s emergency department EHR.
We conducted usability testing of a CDS tool in the emergency department EHR. The CDS tool consisted of the Wells rule for PE in the form of a calculator and was triggered off computed tomography (CT) orders or patients’ chief complaint. The study was conducted at a tertiary hospital in Queens, New York. There were seven residents that were recruited and participated in two phases of usability testing. The usability testing employed a “think aloud” method and “near-live” clinical simulation, where care providers interacted with standardized patients enacting a clinical scenario. Both phases were audiotaped, video-taped, and had screen-capture software activated for onscreen recordings.
Phase I: Data from the “think-aloud” phase of the study showed an overall positive outlook on the Wells tool in assessing a patient for a PE diagnosis. Subjects described the tool as “well-organized” and “better than clinical judgment”. Changes were made to improve tool placement into the EHR to make it optimal for decision-making, auto-populating boxes, and minimizing click fatigue. Phase II: After incorporating the changes noted in Phase 1, the participants noted tool improvements. There was less toggling between screens, they had all the clinical information required to complete the tool, and were able to complete the patient visit efficiently. However, an optimal location for triggering the tool remained controversial.
This study successfully combined “think-aloud” protocol analysis with “near-live” clinical simulations in a usability evaluation of a CDS tool that will be implemented into the emergency room environment. Both methods proved useful in the assessment of the CDS tool and allowed us to refine tool usability and workflow.
Clinical decision support (CDS) tools for pulmonary embolism (PE) diagnosis have been designed and implemented over the past several years with limited success [
A physician’s ability to determine a patient’s risk of disease can sometimes be unclear and can make clinical decisions difficult. CDS tools help providers in their decision making process. These tools have been on the rise in recent years due to their ability to bring evidence-based medicine to point of care. A CDS system (CDSS) is a CDS that is integrated into the EHR and allows physicians to easily utilize the tool effectively. The CDSS incorporates individual patient data, a rule engine, and a medical knowledge base to produce a patient-specific assessment or recommendation of a management plan [
Clinical prediction rules (CPRs) are a type of CDS tool that quantifies the effect an individual patient’s characteristics have toward their diagnosis, prognosis, or likely response to treatment [
A common difficulty with CDS tools is trigger fatigue, when users begin to ignore or override the triggered tool due to a high frequency of alerts [
Emergency medicine physicians across the nation are being asked to improve their resource utilization, while competing with a low tolerance for missed diagnoses. This conflict of interest contributes to emergency department (ED) overcrowding, delay in diagnosis, and unnecessary exposure to radiation. EDs across the nation are backed-up with low risk PE patients waiting for unnecessary computed tomography (CT) scans, while high-risk patients, in need of urgent diagnosis are waiting in the same line. PE patient morbidity and mortality can be improved by timely diagnosis and treatment [
A CPR that is extremely well studied is the Wells score criteria, which enables a physician to predict a patient’s risk of having a PE. The rule has been extensively validated in multiple settings [
Despite successful validation of the Wells score criteria; there has been very limited success with implementation of the rule at the point of care, resulting in underutilization [
Formal usability testing has begun to be considered critical to the EHR adoption and implementation lifecycle [
A recent study emphasized the success of a novel approach to usability testing that combined a “think-aloud” protocol with “near-live” simulations [
“Think-aloud” protocols require users to verbalize their thought process while interacting with a new CDSS tools. For example, specifying why they are clicking on a specific part of the tool and explaining why it is (or is not) helpful. This type of usability testing was specifically well suited for our purpose, due to its ability to identify barriers to adoption and surface level usability issues [
Therefore, we combined this methodology with a “near-live” analysis following the adjustments identified through the first phase of testing. “Near-live” testing allows for a more fluid environment in order to identify further real-life barriers. Historically, “near-live” testing has been used in the engineering world to identify the most effective ways to apply new technologies. However, more recently, it has been documented as a successful methodology for implementing CDSS tools into Health Informatics Systems [
We conducted two rounds of usability testing to identify the optimal way in which to integrate a PE tool into the EHR. This study was the first phase of a larger study looking at the implementation of the Wells CPR in the EHR through a randomized controlled trial. The study took place with emergency room physicians and residents at a large tertiary hospital in Queens, New York. There were four providers that participated during the first phase of the study, and three providers that participated in the second phase of the study. The number of participants involved was based on observations from previous studies where a saturation of feedback was identified at approximately four participants. Therefore, we aimed to recruit approximately four participants in both rounds of testing. A prototype of the EHR was created for the two rounds of usability testing in the Innovation Lab at the Center of Learning and Innovation. Usability data was used to refine and create a production tool. Usability data was used to refine and create a production tool. The PE tool was built as an active CDSS tool that could be triggered by the user during a typical workflow using two different approaches, including patient chief complaint and order entry, the former being upstream versus the latter more downstream (
CDS tool; order entry workflow. PE: pulmonary embolism, ER: emergency room, CT: computed tomography, VQ: ventilation/perfusion, LE: lower extremity, HPI: history of present illness, CDS: clinical decision support, DVT: deep vein thrombosis, SOB: shortness of breath, ROS: review of symptoms, D-dimer: Fibrin split product, MD: medical doctor.
CDS tool; triage nurse workflow. PE: pulmonary embolism, ER: emergency room, CT: computed tomography, VQ: ventilation/perfusion, LE: lower extremity, HPI: history of present illness, CDS: clinical decision support, DVT: deep vein thrombosis, SOB: shortness of breath, ROS: review of symptoms, D-dimer: Fibrin split product, MD: medical doctor.
The four residents who participated in the “think-aloud” phase of usability testing were emergency room residents. Subjects were selected from volunteers to form a convenience sample. Each participant had similar training experience and familiarity with the EHR, ranging between one to three years.
The usability session was conducted at the usability clinic that is associated with our health care system at the Center for Learning and Innovation. Each subject was given thirty minutes to complete four paper cases. The subjects each had two unique cases that had a different level of PE patient risk, varying from low to high. The subjects reviewed two versions of each case; one with the tool popping up at the initial visit through the nurses triage note and the second trigger at the order entry when a CT chest or CTA was ordered.
The subjects were instructed to read each case and enter patient data, develop a progress note, and complete the Wells CPR when it appeared. Using “think aloud” and thematic protocol analysis procedures, scripted simulations of patient encounters with 4 emergency medicine providers were observed and analyzed. Providers were instructed to follow “think-aloud” protocols throughout, which call for them to verbalize all thoughts as they interacted with the mock EHR. The “think-aloud” approach is particularly well suited for studies exploring adoption and implementation issues associated with use of CDS, since it can integrate qualitative and quantitative analyses of provider-decision support interactions.
We collected audio and video recordings of provider’s reactions to the CDS by encouraging them to vocalize their behaviors and thought processes. In addition, all computer screens during the interaction were captured as movie files using the screen recording software. In order to identify how each subject was interacting with the two different CDS tools and how it impacted their workflow, coders grouped facilitators and barriers of each component of the tool. Coders were given a streamlined matrix, training on what to look for, and were instructed to compare and combine thematic codes. For this study, thematic analysis was used in order not only to understand the effectiveness and efficiency of the tool, but also to understand the impact of the tool on the user’s workflow. Following this first phase, we went back through an iterative process of editing the CDS tool from the “think-aloud” feedback.
At the end of the scenario, the subjects were asked for their overall opinion of the tool and it’s positive qualities versus areas for improvement.
The three physicians who participated in the “near-live” clinical scenarios were emergency room residents.
During
Similar to
There were four coding categories that were identified in the first phase of this study: trigger point, calculator, efficiency, and visibility. For trigger point, the subjects felt that the upstream trigger was more effective than a downstream one due to their decision-making process. They felt that if the tool was only triggered by an order entry, their management plan was less likely to change. On the contrary, if the tool was triggered purely on chief complaint, the subjects were more likely to use the tool in order to make their decision. However, a challenge to the upstream trigger point was the lack of all available data in order to complete the tool at that point. When it came to the calculator code, the subjects identified the tool as easy to use and well organized. Furthermore, they felt that in the intermediate cases, when PE diagnosis was unclear, it was better than clinical judgment. The efficacy was determined as being helpful. The visibility of the tool made it clear that there needed to be an option to have the tool on the sidebar of the EHR in order to make it easily identifiable (
Phase I usability coding results.
Code |
|
Example | How it was addressed |
CPR component | Trigger point | Pros Trigger in the beginning helped to frame thoughts around PE diagnosis. Downstream trigger, at order entry, was less helpful due to lack of influence in clinical decision making. Not enough information was attained prior to upstream trigger, which made it hard to complete at that time. |
Upstream trigger was further analyzed. |
|
Calculator | Pros Easy to use and well organized. Wells criteria is well verified and very respected. Too many “clicks” results in “click-fatigue”. |
Employed information technology, IT, to assist with the auto-populating of boxes. |
Usability | Efficiency | Pros Easy to use. Good idea to have a tool. The tool is helpful. It will not trump clinical judgment. Need to ensure that it will be applied to the right subset of patients. |
Need to streamline triggering process to ensure tool is being applied to the necessary population of patients. |
|
Visibility |
Lack of clarity as to where the tool could be found and when it would initiate. |
An option to find the Wells score in the side panel, as a stand-alone tool, is needed. |
General comments | Positive comments |
Well-organized tool. Easy to use. |
The organizational structure of the tool was well received and should be further analyzed in the second round of testing. |
|
Negative comments |
There should be an organized place to place comments and justify a subjects’ clinical thought process. |
Option needed for a text box to appear, where any further comments and/or reasoning can be explained. |
The matrix data from
Similar to
Phase II usability matrix results
Risk level | Participant 1 | Participant 2 | Participant 3 |
High |
|
Agreed with: risk-assessment and order set. | Agreed with: risk-assessment and order set. |
Intermediate | Disagreed with: |
Disagreed with: |
|
Low | Agreed with: risk-assessment and order set. |
|
Agreed with: risk-assessment and order set. |
Following
Therefore, an ideal trigger point that the participants could use was not easily identified. It was clear that an effective trigger point for this tool would need to occur before order onset, but an ideal time was not as clear. This is due to the fast-paced and unpredictable nature of the ED patient flow. If the trigger is placed during ordering, the physician has already chosen the best course of action, has likely informed the patient of their decision, and is less likely to change their management of the patient. However, it was also clear that an upstream trigger point was likely to be too far removed from the physician’s clinical thinking and workflow, and may cause “trigger fatigue”.
The refinements following the first round of usability testing included modification of census trigger to account for patient assessment and the ability to auto-populate from past medical history to address EHR clicking fatigue. From this round, we noted that providers did not use the tool until after they looked at the patient, and in most instances, they had already made a clinical decision before they saw the PE tool.
Upstream versus downstream trigger locations. PE: pulmonary embolism, CDS: clinical decision support.
In both phases of our study, we identified a strong desire for the CDS tool and received positive feedback on the usefulness of the tool itself. Subjects responded that they felt the tool was helpful, organized, and did not trump clinical judgment. However, each round of testing identified clear barriers to integration and areas for improvement. We improved the tool by auto-populating information from the past medical history and identifying ordering bundles to incentivize use. The lack of standardized workflow that the subjects used made identification of a perfect trigger point location extremely difficult, which reinforced the theme to have two trigger locations: one upstream and one downstream to compare the effects on clinical decision making. Our study further demonstrates that usability testing for implementation of CDS tools into the emergency room environment is essential due to the unique challenges that arise.
Although numerous well-validated CPRs exist, few studies have reported significant adoption rates of CPR tools in real-time clinical interactions. A way to address this issue is by integrating CDS tools into the EHR. However, a lack of usability testing prior to their use can result in poor integration within an established clinical workflow [
Due to this gap in literature, we applied the same usability methodology previously applied to the outpatient setting to the emergency room, where the workflow is often chaotic and high stake decisions are often made. This paper summarized the methods and results of the usability testing that we conducted. We hypothesize that conducting usability testing prior to the integration of the PE CDS will increase adoption rates of the tool.
The most important limitation was our ability to simulate a real emergency environment in the simulation center that we have created. However, we instructed subjects to document their encounter and make use of the EHR mirroring the way in which they would do so in their normal clinical environment. Another limitation was the lack of malleability of our EHR system and lag in real-time implementation of subjects’ suggestions due to the technical difficulties in doing so. During the study, we worked closely with an information technology (IT) team to resolve usability issues that we identified during both rounds of usability testing. However, due to the lack of malleability of the EHR system, there were specific elements of the tool that could not be transferred from the prototype EHR to the EHR system utilized by the health care system. An example of this is automatic documentation of the utilization of the tool. An EHR, which is more easily manipulated, would be ideal for this type of study.
This study employed usability testing methodology to analyze the integration of a Wells PE calculator into the emergency room EHR. The first round of testing employed a “think-aloud” approach, which identified numerous opportunities for optimization. By implementing these suggestions into the second round of testing, we were able to increase the usability of the tool. By using a “near-live” approach, we were also able to further identify specific workflow barriers that we were unable to identify in the first round of testing. For example, a desire for a large dismiss button that would allow them to leave the tool incomplete if they chose to do so. Furthermore, they expressed a desire to have the tool as a “suggested next step”, as opposed to mandatory guidelines. Using this methodology in the integration of CDS tools into the ED, we believe we identified bridges that will allow for more seamless integration and adaptation by physicians. The next step in this study is a system wide roll out of the tool in a tertiary care environment.
clinical decision support
clinical decision support system
clinical prediction rule
computed tomography
computed tomography angiography
emergency department
electronic health record
information technology
pulmonary embolism
The authors would like to thank the US Agency for Healthcare Research and Quality for providing us with a grant (5 R24 HS022061) to help conduct this research and to the residents of the LIJ ED for their participation.
None declared.