Developing Implementation Strategies to Support the Uptake of a Risk Tool to Aid Physicians in the Clinical Management of Patients With Syncope: Systematic Theoretical and User-Centered Design Approach

Introduction

Background

Syncope is a prevalent and high-cost problem in emergency departments (EDs) defined as a sudden transient loss of consciousness followed by spontaneous complete recovery [1-4]. It accounted for 15,476,451 ED visits in the United States from 2005 to 2015 [5]. In Canada, there are approximately 160,000 ED visits annually that translate to direct hospital costs of approximately CAD $130 million (approximately US $101 million) [6,7]. Although the cause is often benign, approximately 10% of patients will have serious underlying conditions identified within 30 days [8,9]. Up to half of these conditions will be identified after the initial decision to discharge or admit the patient from the ED, emphasizing the need for supports to improve risk stratification and decision-making in the ED [1,8-10].

The Canadian Syncope Risk Score (CSRS) is a validated risk stratification tool to optimize the accuracy of ED decisions and inform evidence-based clinical actions [9,11]. The tool encompasses the calculation of a risk score from which the following evidence-based practice recommendations for ED disposition are derived: immediate discharge of low-risk patients without subjecting them to further unnecessary testing, consideration for a short course of hospitalization for high-risk patients, and discharge of medium-risk patients with cardiac monitoring and clear information regarding the serious outcome risk. Similar to many decision aids, the uptake of the CSRS is likely to prevent unnecessary hospitalization and improve outcomes for those with underlying conditions through the implementation of a standardized and evidence-based approach to ED syncope management. However, the existence of the CSRS is only part of the solution—physicians must see the value and use the tool in practice to realize the expected benefits.

Health services research shows that many evidence-based practices or interventions fail to demonstrate the anticipated impact as they were not properly implemented, precluding them from achieving a positive impact on clinical practice and patient outcomes [12,13]. Implementation failure might be because of an insufficient understanding of the context in which the implementation process occurs [14,15] or an absence of barriers identification influencing the use of such interventions [16]. The field of implementation science provides systematic approaches and strategies to address the research to practice gap by systematically assessing likely barriers to uptake and identifying implementation strategies to target these barriers [13,17-19].

Objective

The objective of this developmental work was to describe the process of developing a set of implementation strategies to support the use of the CSRS in real-world ED settings to improve syncope management among physicians. This systematic approach not only identifies which strategies are the most appropriate for the target context but also how they should be implemented and operationalized to mitigate the risk of suboptimal implementation or poor uptake among target users [20].

Methods

We followed the process for intervention development outlined by French et al [21], including the first three steps: (1) identifying who needs to do what differently, (2) identifying the barriers and enablers to be addressed, and (3) identifying the intervention components and modes of delivery to overcome the identified barriers.

Study Design

Steps 1 and 2: Identifying Who Needs to Do What Differently and the Barriers and Enablers to Be Addressed

We previously identified that emergency medicine physicians need to change their approach to the assessment of syncope, which would influence their subsequent management among patients. The initial qualitative work identified barriers among 41 physicians across 12 Canadian ED sites to both CSRS use and the adoption of its evidence-based practice recommendations [22]. The most salient barriers identified were workflow issues, concerns about continuity of care, the lack of confidence in the CSRS, and the lack of knowledge and skills around how to interpret and apply the CSRS-related criteria for various patient profiles [22]. The most reported enablers were as follows: legitimacy in the decision rule (CSRS), the evidence of safety and benefit to send the patient home, cardiologists and emergency medicine physicians buy-in, and adequate time with the patient.

The understanding of these barriers and enablers was refined and contextualized with other groups of emergency medicine physicians who participated in workshops as part of a user-centered design (UCD) approach, which is presented in step 3. Essentially, the list of barriers was presented to physicians, and they were asked to react out loud to the following questions: Does this list look complete? Are there other barriers you would like to bring in our attention? and What do you think?

Step 3: Identifying the Intervention Components and Modes of Delivery to Overcome the Identified Barriers

This step was informed by the Behaviour Change Wheel [23], in which we systematically mapped the barriers identified in qualitative work [22] to theoretical determinants, as presented in the Theoretical Domains Framework (TDF) [24] and the Capability, Opportunity, Motivation, and Behavior (COM-B) model [23]. We went through the qualitative findings and looked at physicians-related barriers and facilitators (eg, the lack of knowledge around the eligibility criteria of using CSRS) and matched them with the corresponding theoretical determinant (“knowledge” [TDF] and “capability” [COM-B]). TDF offers a comprehensive lens to look at cognitive, affective, social, and environmental factors [24] that can influence CSRS uptake, whereas COM-B allows for broader categories of determinants.

Once this granular and systematic theoretical understanding of determinants has been completed, we linked them to evidence-based behavior change techniques (BCTs) [25,26], which are the active ingredients or components of an intervention. This step allows to select the most likely techniques to produce the desired change [23,26,27]. From the literature [23,25,26,28], we identified the most effective BCTs that can address each of the behavioral determinants [23,27]. We defined each BCT, how it addressed determinants, and how we could operationalize them. We assessed the feasibility of using these BCTs through peer debriefing (GR, LD, and Marlena Dang Nguyen) and by using the Acceptability, Practicability, Effectiveness, Affordability, Side-effects, Equity criteria [28]. An excerpt of the process of identifying the intervention components and modes of delivery to overcome the identified barriers is presented in Multimedia Appendix 1 [23,25-30]. Selecting those intervention components (also named “implementation strategies”) was the theoretical groundwork that fed the subsequent steps of our developmental process.

We used a UCD approach to validate and refine our understanding of the barriers and enablers and to identify modes of delivery and operationalization. We engaged emergency medicine physicians who were the end users of the CSRS under a collaborative, participatory, and cocreative lens to pursue the parallel goals of maximizing usability in the context of those targeted by the implementation endeavor and tailoring strategies to users’ local contexts while retaining the core components responsible for their effectiveness [18]. This was achieved over a series of 3 qualitative UCD workshops lasting 90 minutes each with 3 groups of emergency medicine physicians. Data collection was performed on the web through synchronous interactions using the Zoom videoconferencing platform (Zoom Video Communications Inc). The workshop-related processes and content are summarized in detail in Textbox 1. Examples of probing questions in the workshop facilitation guide are presented in Textbox 2. GR facilitated the workshops. She did not know the participants before the study. The facilitator (GR) encouraged a “think-aloud” approach to provide insight into participants’ thought processes and gather feedback on which implementation strategies might be useful and why (or why not). Specifically, participants were asked to share their reactions, sentiments, and thought processes in real time. The goal was not to achieve data saturation.

We followed the COREQ (Consolidated Criteria for Reporting Qualitative Studies) [31] to report the qualitative process (Multimedia Appendix 2).

Ethical Considerations

The qualitative and theoretical developmental work was formally reviewed by the institutional authorities at the Women’s College Hospital and was deemed exempt from a research ethics board approval.

Recruitment

Participant recruitment was facilitated by the main developer of the CSRS (VT) through email communication with key informants at 3 ED sites in Ontario, Canada. Individuals who expressed an interest in participating contacted the project lead (GR) and were provided with a study information sheet via email. In each round, we aimed to recruit between 5 and 7 physicians who had different knowledge levels regarding the CSRS (ie, had heard about it and had basic or good knowledge). We used a combination of convenience and purposive sampling to ensure diversity in sex, hospital site (ie, urban academic and nonacademic), and primary language for clinical care (English or French) to capture various perspectives.

Data Analysis

The workshops were audio recorded and the project lead (GR) listened to the complete recordings to partially transcribe key parts of the conversation to shed light on the barriers and facilitators of using the CSRS, as well as the strategies that would be helpful to improve its use. The project lead (GR) and the research assistant (KW) present during the workshops coproduced a preliminary summary of the findings. We performed a qualitative content data analysis according to the predefined objectives for each workshop by using a deductive and inductive process. We used deductive framework coding with broad categories by applying barriers previously identified as deductive codes as well as theory-based strategies. We inductively coded new strategies and operationalization parameters suggested by the participants throughout the conversation. The research team then reviewed the themes and discussed content changes. A summary of the findings and proposed changes was emailed to the participants after each workshop to solicit feedback; validate the emerging insights and operationalization parameters; and seek clarification, if needed (member checking) [32,33].

Results

Participant Characteristics

A total of 14 physicians participated across the 3 workshops and all were aware of the CSRS before participation. The participant characteristics are summarized in Table 1.

The following two themes were identified from the participants’ perspectives: (1) identifying and refining barriers and (2) identifying the intervention components and modes of delivery. The quotes supporting these findings are presented in Table 2.

Theme 1: Identifying and Refining Barriers

Workshop participants validated the following barriers identified in previous study: discomfort using the CSRS, the lack of confidence, the lack of knowledge and skills, and uncertainty around interpretation. Throughout the workshops, physicians highlighted additional barriers, including struggling with how to apply the CSRS recommendations, the inapplicability of the CSRS for some patient clinical presentations, the lack of the CSRS buy-in from the broader medical team (ie, cardiologists and internists), the lack of evidence about the effectiveness of the CSRS tool, and its practice-based recommendations on patient outcomes. The lack of collective buy-in is an important barrier to CSRS use, as physicians have described the importance of all team members. The mapping exercise of linking barriers to theoretical determinants is presented in Multimedia Appendix 1, along with examples on how BCTs (eg, credible source, pros and cons, and instruction on how to perform the behavior) can be operationalized.

Theme 2: Identifying the Intervention Components and Modes of Delivery

Subtheme 2.1: Generating High-Level Strategies and Developing Strategies Prototypes

Participants described that effectively addressing the identified barriers required multiple strategies deployed using various dissemination channels. The need to target a broad audience (ie, emergency medicine physicians, cardiologists, internists, head of department, and nursing staff) with consistent exposure over time (ie, repeat messaging) was emphasized. Participants also highlighted the need to leverage existing structures, including integration of the CSRS into the electronic medical record (EMR), discussing the CSRS at standing educational meetings, and displaying a poster in the workplace environment. There was an agreement that educational meetings could be used to promote general awareness of the CSRS and to encourage a nuanced discussion about its application. However, a range of opinions were expressed on the best format for those educational meetings (eg, combined grand rounds and case-based discussions in small groups). Participants felt that holding combined grand rounds with emergency medicine physicians and specialists (ie, cardiologists and internists) would be a coordinated strategy to address multiple barriers simultaneously. How these educational events are promoted is important to stimulate interest and excitement, including highlighting the credibility of the speaker. Having a journal club with emergency medicine physicians and specialists to review evidence around the CSRS would be useful. Podcasts can be another interesting channel to disseminate knowledge around the CSRS. Participants shared the example of Emergency Medicine Reviews and Perspectives [34], a perceived trusted web-based resource, which is a monthly emergency medicine audio series encompassing continuing medical education. Displaying a poster in the ED was suggested as a helpful visual cue, with participants describing the usefulness of the computed tomography head rule poster [35,36] as an example. Participants also highlighted the need for a local champion that could play multiple roles to model the application and use of the CSRS, to influence uptake among colleagues, speak in educational meetings, to facilitate connection between the clinical and research teams, monitor the implementation process over time, and provide in-person or written feedback.

On the basis of these insights, the research team created 2 prototype posters (Multimedia Appendix 3) and prepared a summary of evidence. The 2 posters encompassed similar content with different displays. In poster 1, information around “For whom the CSRS must be applied” and “When to use CSRS” was highlighted. Poster 2 focused on the proposed course of treatment before the application of the CSRS. In both prototype posters, the CSRS was illustrated along with the 3 risk levels and their proposed practice-based recommendations. In response to participants’ feedback, QR codes were added to the posters: (1) how to use the CSRS, (2) recommendation evidence, and (3) web-based calculator. We prepared a brief summary of evidence to support each practice recommendation for low-risk, medium-risk, and high-risk patients with the intention that physicians feel more confident to use the CSRS and apply the subsequent practice-based recommendations.

The research team also leverages the following existing strategies: educational videos displaying evidence of how the CSRS has been developed and validated and the way to use it, web-based calculators, and email communications used to prompt physicians to use the CSRS and to provide them with positive feedback. The email communication was drawn from a previous pilot study on remote cardiac monitoring as an example to communicate positive feedback on patient impact (ie, an example where home monitoring detected a patient arrhythmia) as well as messaging to remind physicians to use the CSRS.

Subtheme 2.2: Refining and Testing Strategies

Participants discussed the perceived usability, usefulness, and operationalization of the following strategies: educational videos, the poster, the web-based calculator, the summary of evidence, the local champion, and the email communication.

Participants found that the components and features (eg, written summaries, questions, graphics, videos, and links to scientific papers) of educational videos were perceived as useful, and the content was perceived as clear, concise, relevant, and credible. The duration of videos was reasonable if viewed out of the workplace but was too long to be viewed during a shift. It was suggested that a 5-minute video that includes the main information would be an ideal length and would facilitate wider dissemination of the CSRS.

When reviewing the 2 poster prototypes, participants suggested the need to simplify the posters, separating the explicative notes (ie, additional information) from the care pathway, and move those notes as footnotes using a different font (eg, smaller fonts for footnotes) to make the content easier to read. They found poster 2 usable, that is, easy to follow, simple, and appealing. They would use it as a reminder and as a prompt to apply the CSRS, which would be helpful especially at the early stages of the CSRS implementation process in the ED. They would also refer their colleagues to this poster, which is seen as a way of giving credibility to their ED syncope management course of treatment. However, participants identified the following barriers to using such a poster: the risk of poster fatigue, lack of space to display it in their clinical settings, and lack of skills in using QR codes. They suggested to keep only one QR code in the poster, that is, the one related to the web-based calculator.

All participants tested the CSRS web-based calculator [37]. They suggested ways of improving the usability of the web-based calculator: (1) reviewing the wording of some criteria to avoid misleading interpretations, (2) adding a “not drawn” response option to this question “elevated troponin level,” (3) adding access to evidence, and (4) adding access to practice-based recommendations for low-, medium-, and high-risk patients (ie, what to do with the risk score). All participants intended to use the web-based calculator but for different purposes: use in practice and as an educational tool for medical students. Some would use it only if it is integrated into the EMR and will not use it if it is part of a mobile app. Participants would find it useful to discuss in length the summary of evidence in grand rounds or in another type of educational meeting as an initial evidence uptake. Obtaining easy access to evidence was considered important; tying evidence to a web-based calculator and to the EMR would be one way to improve its access.

Finally, email communication with feedback would be useful for emergency medicine physicians to convince them to use the CSRS. Participants had different opinions on how and by whom feedback could be delivered, such as through educational outreach, one-to-one discussion with local champion, and email communication. They would like to receive feedback from the research team (especially the CSRS developers) and a cardiologist within their hospital.

Discussion

Principal Findings

The parameters of these strategies are outlined in Table 3. These strategies will be further developed and deployed as part of a nested process evaluation for a stepped wedge cluster trial.

This work was built on a comprehensive and systematic intervention development process anchored in 1 previous qualitative study and in theoretical mapping of linking theoretical determinants with evidence-based strategies. Furthermore, the contribution of this work is to have involved physicians at different stages to gain insight about the perceived barriers and to test strategies in their context. Key barriers included uncertainty about when and how to apply the CSRS recommendations, the lack of resources (eg, cardiac monitors), the lack of buy-in from the broader medical team, discomfort (hesitancy) using the CSRS, and the lack of evidence about the impact on patient outcomes. Surprisingly, no reference on workload or time constraint was brought up, as is often found in other studies [38-41]. Our findings suggest that physician capability should be a central target of implementation supports, specifically the capability to interpret CSRS-based criteria and apply them across a range of clinical presentations.

Comparison With Prior Work

Training is an evidence-based and frequently used strategy to build physician capability by increasing their skills [23]. This can be operationalized through a variety of mechanisms, including seminars, interactive workshops, and teaching programs such as simulation and training sessions [42,43]. In our study, participants largely referred to educational meetings (eg, combined grand rounds inclusive of all relevant specialties) and educational videos. Skill building can be supplemented by creating increased opportunities to use the CSRS, including integration into the EMR, engaging local champions, displaying posters, and sending email communications to encourage use [40,41,44]. This suite of strategies is commonly used in the ED setting [44] and has demonstrated effectiveness in promoting guideline-adherent care [45].

Achieving collective buy-in across multiple specialty groups (ie, emergency medicine, cardiology, and internal medicine) was highlighted as an essential condition for successful uptake of the CSRS. Although the importance of this broader support is well documented [41,46], including in the area of risk stratification [41], the importance of designing implementation strategies targeting this broader audience (ie, an audience beyond the immediate end user) has been unexplored. Although ED syncope care primarily rests on the shoulders of the emergency physician, support from experts in cardiology, internal medicine, and hospitalists is needed for care of those with suspected or identified serious conditions and further inpatient or outpatient investigations. Physicians rely on their colleagues and professional networks as a unique source of tacit knowledge that serve to either validate initial reasoning or offer alternative approaches [47]. This presents an opportunity to influence uptake through existing channels of social influence that extend beyond the primary setting of interest (in this case, the ED). Strategies may benefit from alignment with the underlying factors that influence patterns of collaboration, including perceived reputational value, experiential information (including personal relationships and visibility), professional identity, and self-awareness of competence [48]. In addition, strategies that target components such as champion or opinion leader, social support, and credible source would be promising ingredients to consider [27].

Similar to the study by Bravo et al [49], our findings highlight a tension between user preferences and scientific evidence and the critical role of triangulating user input in addition to scientific evidence to leverage both sources of knowledge in the design of implementation strategies. Specifically, physician participants identified education in the form of grand rounds as a strategy to address areas of uncertainty and to improve collective awareness of the tool within the interdisciplinary team. Although education effectively increases knowledge and awareness, it is training that effectively builds or strengthens skills [23]. On the basis of physicians’ perceived barrier of not knowing how to apply CSRS, education alone might not be sufficient because it has to do with developing abilities to apply the tool among various patients. In such cases, training would be more suitable. Simply put, each strategy has its own function and mechanisms, allowing it to overcome barriers to using the CSRS and strengthen the facilitators.

Limitations

We used a combination of purposive and convenience sampling to recruit emergency medicine physicians working at 3 different hospitals; therefore, the results may not reflect the experiences of physicians working at other sites. Furthermore, all participants were aware of the CSRS before their participation in the workshops, with most being employed at the same hospital where the CSRS has been piloted. Future studies should explore whether the resulting strategies align with and effectively address the barriers experienced by those who are unaware of the CSRS. Finally, although a comprehensive list of potential BCTs was developed (Multimedia Appendix 1), only a subset of these were prioritized for discussion in the workshops because of feasibility and time constraints. A more comprehensive discussion would have yielded additional strategies to address the identified barriers, and future work should assess whether barriers persist that might be amenable to strategies that were not thoroughly considered. For example, we could operationalize the identification and the preparation of local champions more extensively, and we could target which skills would need to be addressed in a training and how we should impart them (eg, simulation and small-group workshop with demonstration on how to use the CSRS with different patients’ clinical presentations). The provision of performance feedback on the accurate use of the CSRS for risk-stratifying patients by experts (eg, cardiologists and CSRS developers) could also be an avenue to consider.

Conclusions

The ability of the CSRS to effectively improve patient safety and ED syncope management relies on broad buy-in and uptake by physicians. To ensure that the CSRS is well positioned for impact, we identified and developed a comprehensive suite of implementation strategies, including posters, educational meetings (grand rounds), educational videos (with a training component on how to apply the CSRS among various patients), the integration of the CSRS into the EMR, and a web-based calculator to calculate the risk score. These strategies will be evaluated to understand whether and how they are being implemented in practice and whether they are effective in addressing the identified barriers with the objective of improving syncope management in EDs. The next phase of work involves an embedded process evaluation that will provide insight into whether and how this UCD systematic development approach facilitates the ability to effectively target preidentified barriers, physician engagement with the implementation strategies, and broader uptake of the CSRS.