Background: Clinicians often disregard potentially beneficial clinical decision support (CDS).
Objective: In this study, we sought to explore the psychological and behavioral barriers to the use of a CDS tool.
Methods: We conducted a qualitative study involving emergency medicine physicians and physician assistants. A semistructured interview guide was created based on the Capability, Opportunity, and Motivation-Behavior model. Interviews focused on the barriers to the use of a CDS tool built based on Wells’ criteria for pulmonary embolism to assist clinicians in establishing pretest probability of pulmonary embolism before imaging.
Results: Interviews were conducted with 12 clinicians. Six barriers were identified, including (1) Bayesian reasoning, (2) fear of missing a pulmonary embolism, (3) time pressure or cognitive load, (4) gestalt includes Wells’ criteria, (5) missed risk factors, and (6) social pressure.
Conclusions: Clinicians highlighted several important psychological and behavioral barriers to CDS use. Addressing these barriers will be paramount in developing CDS that can meet its potential to transform clinical care.
Clinicians often disregard potentially beneficial clinical decision support (CDS) tools. Extensive study of these tools has shown that their use is associated with a morbidity reduction of 10% to 18%, placing CDS at the top of the spectrum of quality improvement interventions . Improvements in quality of care observed with CDS use [ - ] have been significantly limited by consistently low clinician adoption, estimated at 10% [ , ]. CDS based on Wells’ criteria for pulmonary embolism [ ] serves as an illustration of this phenomenon. Systematic reviews have shown that the use of these criteria decreases ordering of computed tomography (CT) scans by 25% without resulting in additional missed pulmonary emboli (PEs) by clinicians [ ]. However, clinicians have requested the removal of CDS tools based on these criteria, even when local efficacy has been demonstrated [ ].
A systematic review of 58 studies evaluating barriers to clinician adoption of CDS classified these as “CDS specific, organizational, patient and clinician factors” . CDS-specific factors included those that would improve the ease of tool use (ie, minimal mouse clicks, workflow integration). Organizational factors focused on infrastructure and technical issues (ie, having enough computers). Patient factors focused on clinician perceptions of the impact of CDS on the patient-clinician relationship (ie, CDS diminishes the relationship by distracting the clinician). Clinician factors focused on clinician attitudes toward CDS, including a preference for intuitive thought and perception of CDS as a threat to professional autonomy. Clinician attitudes toward CDS, including psychological and behavioral barriers, are not typically addressed during any stage of CDS development although they represent an important barrier to adoption.
Several important publications have detailed the many challenges to CDS reaching its full potential , guiding principles for effective CDS [ ] and barriers to guideline concordant care and successful implementation of CDS [ - ]. However, improved understanding of the psychological and behavioral barriers to clinician use of potentially transformative CDS tools would assist developers in creating highly adopted, high-impact tools. We sought to explore these barriers by using a comprehensive behavioral framework to interview users of a CDS tool based on Wells’ criteria for pulmonary embolism [ ].
We conducted a qualitative study involving emergency medicine physicians (residents and attendings) and physician assistants at two large academic health care facilities in New York. The Northwell Health Institutional Review Board approved this study. Informed consent was obtained for all participants. Participants were recruited by email and presentation at regular faculty meetings. Interviews were conducted between June and September of 2019, and each interview lasted from 30 minutes to 1 hour.
Interview Guide and Behavioral Framework
In-depth interviews focused on the different barriers to use of a CDS tool built based on Wells’ criteria for pulmonary embolism to assist clinicians in establishing pretest probability of PE before imaging. A semistructured interview guide was created based on a comprehensive and parsimonious model of behavior—the Capability, Opportunity, Motivation-Behavior (COM-B) model, which specifies that changing behavior requires changing capability, opportunity, and/or motivation . The COM-B model is at the center of a larger behavioral framework—the Behavior Change Wheel. The Behavior Change Wheel was developed from 19 existing behavioral frameworks and includes 9 intervention functions aimed at addressing deficits in one or more of the conditions described by the COM-B model.
The tool was designed to reduce unnecessary computed tomography pulmonary angiography (CTPA) ordering. Additional details about the design, implementation and evaluation of the tool are available in a previous publication . Emergency clinicians entering any electronic order for the diagnosis of PE (D-dimer, ventilation–perfusion [V/Q] scan, or CTPA) are routed to the tool if they answer “yes” to a dialog box asking, “Are you considering PE?” The tool functions as an expanded order set that allows clinicians to formally calculate pretest probability of PE according to Wells’ criteria. For low-risk patients, it only allows clinicians to order D-dimer laboratory testing and for patients with intermediate or high risk of PE, it allows for D-dimer testing, V/Q scan, or CTPA imaging. At any time, the tool can be dismissed by clinicians and then any order can be placed. The tool was developed using adaptive principles in web and health information technology design, which have been detailed in several previous publications [ - ]. The current version of the tool has been active since January 2016 [ ]; all study participants had previously used the tool in clinical practice.
Thematic saturation was reached after the twelfth interview, with no new insights obtained by the twelfth participant. The COM-B model informed the development of the interview guide, but it was not used to create a priori themes before qualitative analysis. Inductive methods were used to analyze session notes and audio recordings with the COM-B model as a guiding theory. We identified themes using open and then axial coding, and we coded our data accordingly using the qualitative data analysis software NVivo (version 12, released 2018; QSR International Pty Ltd.). Two members (SR and KLD) of the study team, with experience conducting qualitative analysis, coded all sessions. All discrepancies were resolved by consensus.
Interviews were conducted with 5 resident physicians, 5 attending physicians, and 2 physician assistants. Six major barriers to tool use were identified, including (1) Bayesian reasoning, (2) fear of missing a PE, (3) time pressure or cognitive load, (4) gestalt includes Wells’ criteria, (5) missed risk factors, and (6) social pressure ().
|Fear of missing pulmonary embolism|
|Time pressure or cognitive load|
|Gestalt includes Wells’ criteria|
|Missed risk factors|
Clinicians highlighted the belief that the tool was not useful to them because all elements of Wells’ criteria for pulmonary embolism were incorporated into their gestalt. The clinical prediction rule is well known and commonly taught during training in emergency medicine. Fear of missing PE was another major theme identified in our analysis. Patient health consequences were rarely mentioned. Clinicians felt that missed PEs were likely to be less clinically significant and unlikely to result in significant harm to patients, but they worried they still might trigger department quality improvement review or legal action. Time pressure was also highlighted as a major barrier to tool use. Although clinicians denied that cognitive load kept them from using the tool, the majority of clinicians spontaneously mentioned their preference for the pulmonary embolism rule-out criteria (PERC) owing to its simplicity. PERC is validated for use in low-risk patients to rule out PE if eight criteria are negative .
Additional themes included Bayesian reasoning, missed risk factors, and social pressure. Bayesian reasoning reflected some clinicians not recalling that the posttest probability of PE would be impacted by the pretest probability of PE, predicted by the CDS, regardless of the results of the CT scan. Missed risk factors reflected clinicians’ mistrust of the CDS as Wells’ criteria for pulmonary embolism do not explicitly include a few known risk factors for PE. Social pressure reflected many clinicians’ report that other members of the care team, including the patient and their primary care doctor, could influence their decision to not use or not follow the recommendation of the tool.
In this qualitative study of barriers to the use of CDS for the evaluation of PE, participants reported that the CDS tool was not useful to them despite decades of research validating the efficacy of the clinical prediction rule that served as the basis for the tool and our work showing that tool users at our institution improved their CT scan ordering behaviors . Most clinicians felt that they were able to incorporate the elements of the Wells’ criteria for pulmonary embolism into their decision-making without using the tool. The clinical prediction rule, with seven elements, each weighted differently, is complicated enough to make memorization unreliable. There is evidence that clinicians have trouble remembering even simple clinical prediction rules. For example, a study in which clinicians were surveyed about their knowledge of the Ottawa Ankle Rule found that although 89.6% reported using the rule always or most of the time in appropriate circumstances, only 30.9% correctly remembered which four components were part of the rule [ ].
Another major barrier to tool use was fear of missing a PE. In a previous study, surveyed emergency medicine clinicians said that about one-fifth of all imaging studies ordered were medically unnecessary . The main perceived contributors were fear of missing a low-probability diagnosis and fear of litigation. Interestingly, although many clinicians in our study reported this as a barrier, only one knew of any emergency medicine clinician who had ever been sued for a missed PE. The great majority of patients in New York who sustain a medical injury because of negligence do not sue [ , ], and evidence of adherence to known clinical practice guidelines can help clinicians avoid liability [ ]. More importantly, systematic reviews have shown that the use of the Wells’ criteria for pulmonary embolism decreases CT scan ordering by 25% without resulting in additional missed PEs by clinicians [ ]. These facts were not unknown to clinicians in our study, and many volunteered similar statements. However, these facts alone were not enough to address this important psychological barrier to tool use.
Psychological and behavioral barriers, such as gestalt includes Wells’ criteria and fear of missing PE, as well as time pressure or cognitive load are not easily addressed by educational quality improvement interventions. Emergency medicine clinicians are familiar with and believe the Wells’ criteria for pulmonary embolism are useful, as evidenced by a study which surveyed clinicians at our institution . Additionally, the benefits of using the CDS tool, which incorporates these criteria, were reviewed in several academic detailing training sessions for the tool with clinicians before its launch [ ]. Additional educational sessions would not be likely to address the sense for physicians that their gestalt adequately considers Wells’ criteria for pulmonary embolism without referencing them. This is likely to be the case as well for using educational sessions or traditional CDS to reduce fear of missing PE.
Time pressure or cognitive load may be the most difficult to address and an important barrier to the use of CDS in the emergency department. However, clinicians reported that low utility was the driving factor for dismissal, and not cognitive load or time. They also reported the importance of eliminating even a single extra click and a strong preference for PERC owing to its simplicity; however, unlike the Wells’ criteria for pulmonary embolism, it can only be used in low-risk patients. Additionally, emergency medicine may be the clinical specialty with the highest task load and one of the highest cognitive loads . This demanding environment exerts strong pressure on clinicians to find the fastest, safest path forward. In the case of assessment for PE, this often means skipping the CDS and ordering a CTPA—the definitive test to evaluate for PE.
Some of the barriers identified by this study, such as Bayesian reasoning and missed risk factors, might be addressed by simple educational quality improvement interventions. Addressing common knowledge gaps with education—that is, the role of Bayesian reasoning and instances when the rule is not valid—may help to increase adoption rates. A recent study of guideline-discordant CT scans performed to evaluate for PE found that in 39% of these cases, patients had risk factors that were not explicitly incorporated in traditional clinical prediction rules . Building tools with brief instruction manuals may help clarify for clinicians when to use and when not to use these tools. Additionally, although educational quality improvement interventions would be less likely to address barriers such as gestalt includes Wells’ criteria, fear of missing PE, and time pressure or cognitive load, there are several behavioral interventions that might move the needle. For example, tool endorsement by key leadership might increase use, by communicating institutional backing for tool use and mitigating the fear of missing PE. Avenues to address the social pressure barrier would need to be informed by further research, for example, by knowledge of study patients and their preferences.
We have shown how a behavioral model can identify novel barriers to the adoption of a CDS tool. Our findings underscore the importance of addressing the psychological and behavioral barriers to CDS use. Although the field stands to benefit greatly from much anticipated advances in computational capabilities—for example, artificial intelligence, including machine learning—these tools are unlikely to meet their potential to transform clinical care until behavioral barriers to their use are adequately described and addressed.
Our work has several limitations. All clinicians work in the New York City metropolitan area. Both institutions are academic tertiary care centers. Clinicians outside of this geographical area or working in community hospital settings were not included in this study.
In summary, clinicians highlighted several important psychological and behavioral barriers to CDS use. Addressing these barriers will be paramount in developing CDS that can meet its potential to transform clinical care.
This work was supported by grants K23HL145114 from the National Heart Lung and Blood Institute and R01LM012836 from the National Library of Medicine of the National Institutes of Health. Neither source of funding had any role in the design or conduct of the study.
Conflicts of Interest
- Moja L, Kwag KH, Lytras T, Bertizzolo L, Brandt L, Pecoraro V, et al. Effectiveness of computerized decision support systems linked to electronic health records: a systematic review and meta-analysis. Am J Public Health 2014 Dec;104(12):e12-e22. [CrossRef] [Medline]
- Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012 Jul 03;157(1):29-43 [FREE Full text] [CrossRef] [Medline]
- Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003 Jun 23;163(12):1409-1416. [CrossRef] [Medline]
- Bonnabry P, Despont-Gros C, Grauser D, Casez P, Despond M, Pugin D, et al. A risk analysis method to evaluate the impact of a computerized provider order entry system on patient safety. J Am Med Inform Assoc 2008;15(4):453-460 [FREE Full text] [CrossRef] [Medline]
- Roshanov PS, Fernandes N, Wilczynski JM, Hemens BJ, You JJ, Handler SM, et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ 2013 Feb 14;346:f657 [FREE Full text] [CrossRef] [Medline]
- Souza NM, Sebaldt RJ, Mackay JA, Prorok JC, Weise-Kelly L, Navarro T, CCDSS Systematic Review Team. Computerized clinical decision support systems for primary preventive care: a decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes. Implement Sci 2011 Aug 03;6:87 [FREE Full text] [CrossRef] [Medline]
- McGinn TG, McCullagh L, Kannry J, Knaus M, Sofianou A, Wisnivesky JP, et al. Efficacy of an evidence-based clinical decision support in primary care practices: a randomized clinical trial. JAMA Intern Med 2013 Sep 23;173(17):1584-1591. [CrossRef] [Medline]
- Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006 May 16;144(10):742-752 [FREE Full text] [CrossRef] [Medline]
- Lobach D, Sanders GD, Bright TJ, Wong A, Dhurjati R, Bristow E, et al. Enabling health care decisionmaking through clinical decision support and knowledge management. Evid Rep Technol Assess (Full Rep) 2012 Apr(203):1-784. [Medline]
- van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13(2):138-147 [FREE Full text] [CrossRef] [Medline]
- Wells PS, Anderson DR, Rodger M, Stiell I, Dreyer JF, Barnes D, et al. Excluding pulmonary embolism at the bedside without diagnostic imaging: management of patients with suspected pulmonary embolism presenting to the emergency department by using a simple clinical model and D-dimer. Ann Intern Med 2001 Jul 17;135(2):98-107. [CrossRef] [Medline]
- Wang RC, Bent S, Weber E, Neilson J, Smith-Bindman R, Fahimi J. The impact of clinical decision rules on computed tomography use and yield for pulmonary embolism: a systematic review and meta-analysis. Ann Emerg Med 2016 Jun;67(6):693-701.e3. [CrossRef] [Medline]
- Drescher FS, Chandrika S, Weir ID, Weintraub JT, Berman L, Lee R, et al. Effectiveness and acceptability of a computerized decision support system using modified Wells criteria for evaluation of suspected pulmonary embolism. Ann Emerg Med 2011 Jun;57(6):613-621. [CrossRef] [Medline]
- Moxey A, Robertson J, Newby D, Hains I, Williamson M, Pearson SA. Computerized clinical decision support for prescribing: provision does not guarantee uptake. J Am Med Inform Assoc 2010 Jan 01;17(1):25-33. [CrossRef]
- Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, et al. Grand challenges in clinical decision support. J Biomed Inform 2008 Apr;41(2):387-392 [FREE Full text] [CrossRef] [Medline]
- Campbell R. The five "rights" of clinical decision support: CDS tools helpful for meeting meaningful use. J AHIMA 2013 Oct;84(10):42-7 (web version updated February 2016) [FREE Full text] [Medline]
- Sheehan B, Nigrovic LE, Dayan PS, Kuppermann N, Ballard DW, Alessandrini E, Pediatric Emergency Care Applied Research Network (PECARN). Informing the design of clinical decision support services for evaluation of children with minor blunt head trauma in the emergency department: a sociotechnical analysis. J Biomed Inform 2013 Oct;46(5):905-913 [FREE Full text] [CrossRef] [Medline]
- Novak LL, Holden RJ, Anders SH, Hong JY, Karsh BT. Using a sociotechnical framework to understand adaptations in health IT implementation. Int J Med Inform 2013 Dec;82(12):e331-e344 [FREE Full text] [CrossRef] [Medline]
- Melnick ER, Shafer K, Rodulfo N, Shi J, Hess EP, Wears RL, et al. Understanding overuse of computed tomography for minor head injury in the emergency department: a triangulated qualitative study. Acad Emerg Med 2015 Dec;22(12):1474-1483 [FREE Full text] [CrossRef] [Medline]
- Richardson S, Cohen S, Khan S, Zhang M, Qiu G, Oppenheim MI, et al. Higher imaging yield when clinical decision support is used. J Am Coll Radiol 2020 Apr;17(4):496-503 [FREE Full text] [CrossRef] [Medline]
- Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci 2011 Apr 23;6:42 [FREE Full text] [CrossRef] [Medline]
- Press A, Khan S, McCullagh L, Schachter A, Pardo S, Kohn N, et al. Avoiding alert fatigue in pulmonary embolism decision support: a new method to examine 'trigger rates'. Evid Based Med 2016 Dec;21(6):203-207. [CrossRef] [Medline]
- Press A, McCullagh L, Khan S, Schachter A, Pardo S, McGinn T. Usability testing of a complex clinical decision support tool in the emergency department: lessons learned. JMIR Hum Factors 2015 Sep 10;2(2):e14 [FREE Full text] [CrossRef] [Medline]
- Khan S, Richardson S, Liu A, Mechery V, McCullagh L, Schachter A, et al. Improving provider adoption with adaptive clinical decision support surveillance: an observational study. JMIR Hum Factors 2019 Feb 20;6(1):e10245 [FREE Full text] [CrossRef] [Medline]
- Khan S, McCullagh L, Press A, Kharche M, Schachter A, Pardo S, et al. Formative assessment and design of a complex clinical decision support tool for pulmonary embolism. Evid Based Med 2016 Feb;21(1):7-13. [CrossRef] [Medline]
- Kline JA, Mitchell AM, Kabrhel C, Richman PB, Courtney DM. Clinical criteria to prevent unnecessary diagnostic testing in emergency department patients with suspected pulmonary embolism. J Thromb Haemost 2004 Aug;2(8):1247-1255 [FREE Full text] [CrossRef] [Medline]
- Brehaut JC, Stiell IG, Visentin L, Graham ID. Clinical decision rules "in the real world": how a widely disseminated rule is used in everyday practice. Acad Emerg Med 2005 Oct;12(10):948-956 [FREE Full text] [CrossRef] [Medline]
- Kanzaria HK, Hoffman JR, Probst MA, Caloyeras JP, Berry SH, Brook RH. Emergency physician perceptions of medically unnecessary advanced diagnostic imaging. Acad Emerg Med 2015 Apr;22(4):390-398 [FREE Full text] [CrossRef] [Medline]
- Localio AR, Lawthers AG, Brennan TA, Laird NM, Hebert LE, Peterson LM, et al. Relation between malpractice claims and adverse events due to negligence. Results of the Harvard Medical Practice Study III. N Engl J Med 1991 Jul 25;325(4):245-251. [CrossRef] [Medline]
- Studdert DM, Thomas EJ, Burstin HR, Zbar BI, Orav EJ, Brennan TA. Negligent care and malpractice claiming behavior in Utah and Colorado. Med Care 2000 Mar;38(3):250-260. [CrossRef] [Medline]
- Samanta A, Mello MM, Foster C, Tingle J, Samanta J. The role of clinical guidelines in medical negligence litigation: a shift from the Bolam standard? Med Law Rev 2006;14(3):321-366. [CrossRef] [Medline]
- Richardson S, Khan S, McCullagh L, Kline M, Mann D, McGinn T. Healthcare provider perceptions of clinical prediction rules. BMJ Open 2015 Sep 02;5(9):e008461 [FREE Full text] [CrossRef] [Medline]
- Harry E, Sinsky C, Dyrbye LN, Makowski MS, Trockel M, Tutty M, et al. Physician task load and the risk of burnout among US physicians in a national survey. Jt Comm J Qual Patient Saf 2021 Feb;47(2):76-85. [CrossRef] [Medline]
- Simon E, Miake-Lye IM, Smith SW, Swartz JL, Horwitz LI, Makarov DV, et al. An evaluation of guideline-discordant ordering behavior for CT pulmonary angiography in the emergency department. J Am Coll Radiol 2019 Aug;16(8):1064-1072 [FREE Full text] [CrossRef] [Medline]
|CDS: clinical decision support|
|COM-B: Capability, Opportunity, and Motivation-Behavior|
|CT: computed tomography|
|CTPA: computed tomography pulmonary angiography|
|PE: pulmonary embolism|
|PERC: pulmonary embolism rule-out criteria|
Edited by A Kushniruk; submitted 15.10.20; peer-reviewed by E Melnick, S Sarbadhikari; comments to author 24.11.20; revised version received 05.02.21; accepted 05.04.21; published 04.08.21Copyright
©Safiya Richardson, Katherine L Dauber-Decker, Thomas McGinn, Douglas P Barnaby, Adithya Cattamanchi, Renee Pekmezaris. Originally published in JMIR Human Factors (https://humanfactors.jmir.org), 04.08.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on https://humanfactors.jmir.org, as well as this copyright and license information must be included.