In phase 1 of our study, 2 independent reviewers completed review of academic literature on MEDLINE to build a list of usage categories of EHR. The reviewers also searched gray literature through web-based hand searches for topics related to information systems in acute care settings. After an iterative review of literature relating to both emergency medicine settings and EHR, 5 usage categories were developed inductively by the 2 reviewers. The findings were discussed with a working group comprised of 4 investigators with expertise in emergency medicine, health systems, and health informatics. The working group came to an agreement about 5 proposed key usage categories that were inputted into phase 2 to narrow the focus of discussion.

Phase 2 used Delphi methods that involved sequential rounds of survey and data dissemination to experts in both emergency medicine and information systems regarding their perspectives on each of the 5 usage categories. Recruitment of expert panelists was done through purposive sampling beginning with 4 investigators identifying candidates with expertise in both the clinical environment of the ED and health informatics at 6 tertiary- and quaternary-care centers across southwestern Ontario, including 3 level 1 trauma centers. Subsequently, the identified candidates were also invited to provide information on other potential informants. In total, 12 expert panelists were recruited across several hospital systems with extensive experience in both emergency medicine and health information systems. The panelists were spread across 3 separate disciplines (7 of 12 in emergency medicine, 3 of 12 in pediatrics emergency medicine, and 2 of 12 in general internal medicine). Several panelists held multiple leadership roles in their departments, with 4 of 12 acting as either chief or deputy chief, 7 of 12 acting as department lead across roles in quality and safety, virtual care, artificial intelligence and machine learning, and quality improvement. Several panelists also performed adjacent clinical duties with 4 of 12 serving as Trauma Team Leaders. Two panelists also fulfilled C-level positions at their respective hospital systems for roles in medical informatics. All panelists were associated with the University of Toronto in teaching and academic roles.

The Delphi study was conducted in 3 rounds of surveys [11]. Survey administration was conducted using the Research Electronic Data Capture (REDCap 12.0.29) tools hosted at the University of Toronto [19,20]. To reduce bias in both survey responses and response analysis, the identity of all panelists was kept anonymous through the Delphi rounds. Panelists and investigators were unaware of the identity of panelist’s responses and panelists were not aware of the identity of other members of the Delphi panel until the conclusion of the study. The analysis of outputs from each round was conducted by 2 independent reviewers and consensus was established before circulation of findings to panelists between each round.

The first-round survey involved qualitative information gathering through free-text responses. Free-text responses were analyzed using NVivo (NVivo Version 12). First, responses were coded deductively, using usage categories defined in Phase I of the study. Second, sentiment coding was performed by NVivo’s sentiment analysis with manual adjustment and necessary recoding based on consensus by the 2 independent reviewers. Outputs were circulated to panelists for review. The second-round survey gathered quantitative information on the perceived importance of first-round outputs using Likert scales and qualitative free-text responses about areas of disagreement from first-round responses. The quantitative outputs from the second-round survey were analyzed using Microsoft Excel (MSO Version 2205; Microsoft Inc) to generate descriptive statistics around measured variables and the qualitative outputs from the second-round survey were circulated to the panelists [21]. The third-round survey focused on establishing a ranked list of priorities based on the second-round outputs with the highest perceived importance resulting in a ranked list of priorities for each usage category.

Phase 2 received the approval of the Research Ethics Board through the University of Toronto (protocol #00040996).

Overall, it was found that panelists preferred that current EHRs improve on existing capabilities before trying to tackle potential disruptive innovations [22]. Panelists specifically listed and ranked digital ambient scribes, which process information from a patient–physician interview into a note in an attempt to reduce documentation burden, and auto-population of documentation from other sources of clinical information, lower than basic functionality such as multiauthor documentation and support for documentation of other forms of media. As strength, it was found that panelists thought that EHRs have streamlined the collation and standardization of information. A limitation of current information input capabilities of EHRs is the lack of support for multiauthor documentation, increasing the need to repeatedly gather, and document redundant information that has already been collected by other members of the patient’s care team. This drives documentation burden and creates inefficiencies.

It was largely believed by panelists that human factors limit the implementation of digital health tools such as machine learning algorithms that provide clinical decision support, as opposed to the technical capacities of the current EHRs. Furthermore, the priorities list shows that panelists prioritize tools supporting clinicians in acute care settings such as identifying high-risk patients as opposed to pulling previous information from other sources such as previous charts or clinical portals. Panelists mostly expressed that EHRs have streamlined the ability to conduct repetitive, previously tedious tasks. However, they state that innovation requires large amounts of coordination and health human resources, so while the potential may exist for implementation, there may not be the current appetite or means to sustain this change.

The priorities of end users in this category saw 2 sentiments of thought, which first may seem conflicting. On the one hand, there was an interest in having increased customizability options within ED EHRs, such as the enablement of customization of quick picks and inbox management. However, there was also the argument for adaptation on the part of the end user to the features and limitations of the EHR. Overall, panelists believed that EHRs have increased standardization of care delivery through order sets that are vetted by central decision support teams, ensuring that orders are up to current care standards. However, in their current form, EHRs are limited in the customization options that they provide for their end users, even with respect to personal workflow features such as inbox task management, or “look and feel” customizations such as the layout of a given dashboard.

Panelists again prioritized basic functionality (ie, discharge planning, interdisciplinary communication) as opposed to disruptive innovation. Although EHRs have increased ease of collaboration among teams in the ED through collation of documentation from triage, panelists still raised concerns around the limitations of interoperability between hospital systems and other systems such as primary care EHRs. Additionally, even within a single-hospital system, it was found to be difficult to communicate with other services that did not use the same EHR or charting method (ie, different clinical systems or paper charting).

Overall, panelists express that there was limitation with fluid access and usability of information. An undeniable strength of the EHR is that it has augmented the ability to collect, store, and access structured data. However, panelists identified that the ability to access the data in a meaningful way is still limited due to the format of stored data. Although it is possible to access volumes of information, the standardization of information input is lacking, such that any information sought for research purposes will still require manual recoding. Suggestions in this realm included improving drop-down menus to provide standardization of documentation input.

The key usage categories developed in our investigation and the panelists’ priorities determined by Delphi outputs span several steps of a patient’s journey through the ED (Figure 1). These priorities highlight the balancing act that must occur in each usage category with the development and deployment of ED EHRs. With respect to information input, support for multiauthor documentation helps to reduce redundancy of information gathering and input, and support for innovations helps reduce documentation burden. With respect to digital health tools, improved governance structures could support the development and deployment of innovations that may aid in decision-making. With respect to usability, an optimized EHR for the ED would have customizability options for workflow and maintain strong standardization for deployment of care, such as order sets. With respect to clinical workflow support for communication beyond the hospital helps to ensure efficient and safe patient discharges, while consolidated information systems ensure efficient access to conducted investigations. With respect to research and data analytics, improved accessibility allows for more contribution from end users with respect to the development of new knowledge and useful clinical insights.

In Gawande’s [46] article titled, “Why Doctors Hate Their Computers,” Gawande writes of EHRs: “I’ve come to feel that a system that promised to increase my mastery over my work has, instead, increased my work’s mastery over me.” His assertion mainly centers around the collection of large amounts of unused information from a patient–physician encounter, which drives documentation burden and decreases patient–physician interaction time. Previous studies have estimated that the ED physicians may spend as much as 25% of the total time caring for a single patient on documentation [47]. Aligned with the previous literature and clinical experiences of documentation burden, Gawande highlights a key issue where EHRs can decrease efficiency and become a burden rather than a valuable tool.

These concerns are aligned with the findings from this investigation, with panelists broadly prioritizing functionality over disruptive innovation, and issues such as interoperability and the reduction of documentation burden being prioritized across several usage categories. For example, with respect to information input, support for multiauthor documentation and picture integration was prioritized over features such as digital ambient scribes or population from past documentation. Another example is seen in panelists’ priorities with respect to clinical workflow, where panelists prioritized discharge communication methods over auto-population of patient information from previous documentation. Panelists were sampled from a variety of care settings employing several different EHRs at each site, suggesting that no single EHR vendor comprehensively captures the priorities identified in this investigation.

By examining the discrepancies between the identified priorities of panelists and the qualitative responses of strengths and limitations, it is possible to identify areas for impactful improvements. For example, with respect to digital health tools, streamlined governance structures were identified as both a top priority (Textbox 1) and listed as a limitation (Table 2). Another usage category that demonstrated this was in research and data analytics, where panelists identified streamlined governance structures and increased role-based access as priorities (Textbox 1) and identified privacy as a limiting factor for gathering information (Table 2). Integrating this information identifies areas of high priority and can potentially inform prioritization of where system administrators can best optimize their own EHRs or build evidence-informed criteria in future acquisitions.

Compared to the deployment of Delphi methods in other emergency medicine clinical questions, the modifications to the process of this investigation optimized for depth of discussion in defined usage categories. The specific modification to the traditional process entailed defining the 5 usage categories through literature review which subsequently served as inputs to the Delphi model. Other investigations either integrate the literature review as one of the 3 traditional rounds or rely on free-text responses as a means to providing a focus of discussion [16,17]. A trade-off of the selected modification is that it prevents panelists from suggesting their own mental schema of usage categories of EHRs; however, this trade-off was made to achieve a deeper understanding of priorities within discrete categories. An additional benefit of a preliminary literature review is that focused discussion ensured concrete outputs from each round, which may have contributed to the complete retention of panelists across the 3 rounds of the Delphi process. Overall, through a preliminary literature review and a Delphi process with narrow targets based on prior inputs, the modified Delphi method strikes an appropriate balance between breadth and depth in the examination of ED EHRs.

One potential limitation to this study is the generalizability of findings. Panelists are familiar with both ED care settings and health informatics in tertiary-care hospitals in southern Ontario, all with enterprise-wide deployments of their hospital EHR. This may lead to panelist-specific prioritization of other clinically adjacent activities such as academic research or data organization. Subspecialty interests may introduce additional variance to captured perspectives. Furthermore, this investigation focused on capturing the perspectives of physicians as the end user, which does not capture the perspectives of other disciplines that engage with an ED EHR.

Improving EHRs to effectively meet the unique priorities of the ED demands a thorough understanding of the priorities of end users. A modified Delphi approach allows an in-depth analysis of perspectives of expert panelists in discretely defined usage categories. Capturing the perspectives of an expert panel from tertiary and quaternary care centers across Southwestern Ontario and served by diverse EHR vendors, the findings of this study highlight end-user prioritization of functionality over disruptive innovation. At a provider level, these findings will lead to meaningful reflection and discussions with department leadership about how an EHR can fit local needs. At an institution level, these findings will have implications for choosing future EHRs and adaptation of existing systems. At a developer level, these findings will have further sensitized developers to the preferences of end users in high-acuity settings. The future steps in discussions around EHR improvement should involve gathering the perspectives of allied health professionals who also engage with EHRs and with patients as they are the beneficiaries of improvements to information systems. Furthermore, comparison of perspectives gathered in the ED to perspectives from other areas of the hospital would establish commonalities, common pain points, and enhance our understanding of the information system preferences of end users.