Hypoglycemia is a common occurrence in hospitals and has been linked to poor clinical outcomes and increased mortality [1,2]. Patients with and without diabetes can experience acute hypoglycemic episodes in the hospital, which may result in outcomes ranging from mild distress and patient dissatisfaction with cardiac ischemia, arrhythmias, loss of consciousness, stroke, seizures, and coma [1,3,4]. Of the 8 million and rising number of patients with diabetes who are admitted to hospitals in the United States annually, up to 25% may experience a hypoglycemic episode during hospitalization [5]. Approximately half of these episodes can be explained by an underlying illness, such as severe sepsis, renal failure, liver failure, or malignancy [3]; however, the remaining half are iatrogenic in nature, usually resulting from insulin treatment [6].

Insulin remains the recommended therapy for most patients with diabetes during hospitalization [7,8]. Recent data from the Medicare Patient Safety Monitoring System found that glucose-lowering medications were associated with the highest rates of adverse outcomes for all drugs used in the hospital [9]. As insulin accounts for the vast majority of hypoglycemic events [10,11], the Joint Commission and the Institute for Safe Medication Practices have designated it a high-alert medication [12]. Insulin is typically administered as a continuous infusion for critically ill patients in the intensive care unit (ICU) and as subcutaneous injections in noncritically ill patients in general medical or surgical wards. In contrast to the ICU setting where insulin adjustments are driven by nurse-managed protocols, insulin titration in the non-ICU setting is prescriber-driven and requires evaluation of a complex set of clinical, laboratory, and pharmacological parameters.

Clinical decision support (CDS) tools in electronic medical record (EMR) systems have been increasingly used in the United States and have been shown to improve the processes of care and clinical outcomes [13-15]. These tools have been used to alert clinicians, suggest diagnostic or treatment recommendations, and provide contextually pertinent information to optimize care for a wide variety of indications in the hospital setting, ranging from sepsis to acute kidney injury [16-18]. It stands to reason, then, that CDS could be used to improve care of patients vulnerable to hypoglycemia. Prediction models using large EMR data sets have been developed to trigger alerts in patients at risk of hypoglycemia [19-22]. Several studies have found that these predictive models can decrease the rate of inpatient hypoglycemic episodes, with one review finding that it could decrease the incidence of severe hypoglycemia by up to 68% [19,23].

At our institution, we have several existing CDS tools to guide clinicians in selecting a safe and effective initial insulin dosing regimen, including a mandatory subcutaneous insulin order set and an optional insulin CDS tool [24]. Although our subcutaneous insulin CDS tool was derived from evidence-based basal-bolus insulin dosing protocols [25], even universal use of this tool would not be expected to prevent all iatrogenic hypoglycemic events in the hospital because of the impact of acute illness and other clinical factors on glucose homeostasis (eg, nutritional status, renal function, and steroid doses).

To address the need for real-time hypoglycemia risk detection, we recently developed a machine learning algorithm using EMR data that accurately predicts iatrogenic hypoglycemia in rolling 24-hour windows following each blood glucose reading during hospitalization [22]. In planning the translation of this machine algorithm into a real-time informatics alert, we sought to obtain feedback from inpatient clinicians who are responsible for the day-to-day management of blood glucose in the hospital. The objective of this study is to obtain stakeholder input regarding the design of a real-time hypoglycemia informatics alert for use in a hospital setting.

This was a mixed methods study that integrated quantitative and qualitative data. The initial stage consisted of an electronic survey (administered in February 2019) that was sent to physicians and advanced practice providers at 2 academic medical centers located in Baltimore, Maryland. The second stage consisted of 2 separate focus groups (conducted in September 2019) with participants recruited from the pool of survey respondents, the goal of which was to expand further on the answers to the survey and identify common themes. This study was approved by the institutional review board at the Johns Hopkins School of Medicine. For the electronic survey, consent was implied from respondent completion, and written informed consent was obtained from all participants in the focus group sessions.

To evaluate the key features and functionality of a hypoglycemia risk alert, we developed a 21-item electronic survey that was administered through SurveyMonkey (SVMK Inc) via an embedded email hyperlink (Multimedia Appendix 1). The survey required approximately 7-10 minutes to complete and was sent to hospital-based clinicians involved in glucose management, including medical and surgical residents, hospitalists, surgical advanced practice providers, and inpatient diabetes nurse practitioners (NPs). To encourage participation, the email invitation with the survey link was sent directly by residency program directors (medicine, surgery, neurology, obstetrics/gynecology [OB/GYN]) to all residents in their programs, and by program leaders in the hospitalist, surgical advance practice provider groups, and inpatient diabetes management service. Thus, the total number of recipients who received the survey link (and hence the response rate) was not known by the study investigators.

Survey questions were developed using the Agency for Healthcare Research and Quality Five Rights of effective CDS as a guiding framework: the right information to the right person, in the right interventional format, through the right channel, and at the right time in the workflow [26]. Right information refers to the content and presentation of information to the end user (accuracy, estimated risk and reasons for predicted risk, and recommended action). Right person refers to the member of the health care team who is most appropriate to receive and respond to the alert and the method of identifying this individual. Right format refers to the type of CDS used to address the clinical scenario and right channel refers to the platform for communicating the alert. We considered several formats or channels within and outside our EMR system (Epic Version 2018, Epic Systems Corporation), including the best practice advisory (BPA), glucose management print group report, patient system lists, patient header, InBasket messaging, and text messaging. A description of each of these formats or channels is provided in Table 1, with example screenshots in Figure 1. Finally, right time in the workflow, a critical component of successful CDS interventions, refers to the time when the alert is presented to the end user in their usual clinical workflow to minimize disruption and achieve desired action. Multimedia Appendix 2 categorizes the survey questions according to the Five Rights topics.

To gain a more in-depth understanding of the perceptions, preferences, and perceived barriers to a hypoglycemia informatics alert, we conducted 2 focus group sessions, each consisting of 6 different participants, after the results of the survey were analyzed. Hospital-based physicians, NPs, and PAs were eligible for participation. Participants who responded to the initial survey expressing interest in further research related to the topic were invited to the focus group sessions. No participant identifiers were used during the session, and all participants were referred to by a given letter at the start of the session.

The focus group sessions lasted 60-90 minutes and were led by an experienced focus group moderator (LLB). An institutional review board approved–approved structured interview guide was used (Multimedia Appendix 3), with questions designed to delve into survey results and reasoning behind responses. The principal investigator (NM) was also presented to address or clarify any participant questions during the session. The sessions were audio-recorded and professionally transcribed. The investigators further reviewed and corrected the transcriptional errors. The participants were provided with food during the session and a US $100 gift card.

Descriptive statistics were used to summarize the characteristics of the survey respondents and focus group participants. For binary, multiple-choice responses and Likert response data, the number and percentage of responses were provided. For rank items, the average ranking was calculated as the weight of the ranked position. MAXQDA (Verbi Software, 2019) was used for qualitative analysis and thematic coding of data from the focus group sessions. We used a combination of deductive and inductive methods in our analyses [27]. Before coding the data, we identified an initial set of deductive codes centered on the CDS Five Rights framework. Close reading of the transcripts then informed the development of inductive codes, reflecting the ground, or participants’ experiences. When the code structure was considered final (ie, no new concepts were apparent), one researcher independently applied the finalized code structure and synthesized the data into patterns (ie, a cohesive category of responses found across our participants) and themes (ie, a broad concept or topic that aggregates patterns).

Table 2 shows the characteristics of the survey respondents and participants in the 2 focus group sessions. There were a total of 102 survey respondents, most of whom were internal medicine (IM) physicians, with fairly even representation from trainees and faculty or staff. Focus group 1 comprised 6 NPs (of whom 4 were inpatient diabetes specialists) and Focus group 2 comprised 6 physicians (half of whom were medicine residents).

Figure 2 summarizes the responses to the survey questions related to the importance of the problem and the perceived benefit of the proposed alert. There was unanimous agreement that preventing insulin-related hypoglycemia in hospitalized patients is an important priority, and 47% (48/102) either agreed or strongly agreed that preventing insulin-related hypoglycemia in the hospital is challenging. Nearly all providers reported that they reviewed blood glucose data on insulin-treated patients and almost all providers felt that they recognized glycemic patterns indicative of a need to adjust insulin doses. Accordingly, 82.3% (84/102) of the respondents felt confident in their ability to safely adjust their insulin doses. Over two-thirds felt that a real-time hypoglycemic alert would be beneficial.

Table 3 summarizes the survey responses related to the proposed information alert. The top 3 preferred formats were an alert indicator in the patient header, secure text message, and alert prompt displayed in the existing glucose management report. The least desired formats were an Epic InBasket message, a patient system list, and a pop-up BPA. In terms of information included in the alert, 78.4% (80/102) of the respondents indicated that they would want to know the specific reasons why an individual patient is predicted to be at risk based on the prediction model; 64.7% (66/102) of the respondents wanted the alert to categorize the predicted hypoglycemia risk into low, medium, and high levels; and 62.8% (64/102) wanted the alert to give an estimated probability of hypoglycemia. A minority of respondents (21/104, 23.5%) indicated that they would want the alert to hyperlink to an actual prediction model, and 16.7% (17/102) indicated that they would want the validated accuracy of the model displayed in the alert. With respect to accuracy, there was a wide range of acceptable sensitivity and specificity thresholds among respondents, with most considering 70% and above to be acceptable for both.

The vast majority of the respondents (90/102, 88.2%) indicated that they would want the alert to provide some recommended action, and 59.8% (61/102) of respondents suggested that there should be an option acknowledge the alert or ignore or override the alert. Other features of the alert desired by approximately half or less of the respondents included direct hyperlinks to our subcutaneous insulin order set (54/102, 52.9%) and subcutaneous insulin decision support tool (29/102, 28.4%) and the ability to directly consult endocrinology or inpatient diabetes management service (43/102, 42.2%). Although some respondents felt that the ability to easily consult endocrinology or diabetes services was important, most (85/101, 84.2%) felt that the alert should not automatically trigger a consult.

Regarding the right person to receive the alert, 84.3% (86/102) of respondents indicated the person listed as first call in the EMR should be notified. In addition to the first call provider, 46.1% (47/102) of respondents indicated that the patient’s nurse should also be notified. With respect to the workflow, 46.1% (47/102) of respondents indicated they would want to be notified as soon as the hypoglycemia risk was identified, whereas 26.5% (27/102) preferred the same time every day and 23.5% (24/102) preferred being notified only when entering the chart of a relevant patient.

The core themes discussed during the focus groups centered on the CDS five rights format, with the aim of receiving feedback on how a functional alert system could best suit front-link clinicians at a major hospital, as well as anticipated barriers and challenges. Representative quotes illustrating each key theme are presented in Table 4.

Accuracy was a common theme. According to the surveys, most of our participants wanted the alert to be at the very least 70% sensitive and specific. One participant noted as follows:

Participants raised concerns that if the specificity was low at launch, the alert would go off repeatedly and would be ignored. Many noted that similar alert systems have been deployed at our institution before validation of accuracy; as a result, many of the alerts were initially unsuccessful at gaining end user buy-in and changing behavior. To avoid this, one participant noted that the algorithm should be continuously adjusted to increase accuracy over time.

A common theme among focus group participants was the importance of defining specific reasons that triggered an alert. Participants felt that sharing this information would present a learning opportunity for end users and could indirectly modify behavior in their care of future patients. Participants also voiced the concern that there should be an option to report the alert if it is inaccurate:

Finally, echoing the results of our survey, the focus group participants emphasized their desire to have an alert to recommend the appropriate action. The tendency of alert systems to interrupt workflow without recommending an appropriate action leads to clinicians feeling overwhelmed. Participants noted that the best part of the other alert systems was the ability to help with workup and management. One participant noted as follows:

There were mixed opinions among the focus group participants as to the right person to be notified of the alert. Many believed that contacting the individual or service listed as first call in the EMR would be the most sensible approach. However, many voiced the concern that the only way a first-call system would be efficient was if the EMR was accurately updated, which may not always be the case. Other participants voiced that the right person to be alerted should always be a nurse. One participant noted that with other alert systems, the nurse is responsible for triaging the alert and determining whether contacting the provider is required:

There was substantial heterogeneity in preferences regarding the alert format, and our discussion focused on the pros and cons of each format. Focus group participants noted that an alert indicator in the patient header would be a very suitable place for a hypoglycemia alert; however, concerns were raised that the patient header is already a very crowded space:

Others really valued the idea of using the glucose management report, which is a tabular report summarizing insulin and glucose data in a temporal way that facilitates pattern recognition and insulin dose adjustments but cautioned that this passive approach may not achieve the desired action. A suggestion raised during the focus groups session that was not considered in the electronic survey was made to place an alert symbol next to a patient’s glucose laboratory result to notify them of impending risk; this alert would be distinct from an abnormal laboratory result value to distinguish predicted risk from overt hypoglycemia. Participants felt that a results flag would be the most effective in increasing situational awareness:

A consensus was also reached that BPAs were not ideal for an alert system, as they were noted to be very interruptive by some of the participants:

The Epic InBasket format was dismissed by most participants, as this is not a format routinely used for communication in the inpatient setting:

The CORUS text messaging system was the unanimously desired channel for receiving the alert. All participants agreed that it was the quickest and most efficient way to be notified. A few even mentioned that they were too heavily reliant on CORUS, but noted it was the appropriate app to get the staff’s attention:

The majority agreed that the alert should be direct and in real time to allow for appropriate action. One concern that many had was the fact that the alert might go off in a time when the staff member could not respond. One participant offered the suggestion of a snooze button to address this concern:

The focus group sessions elicited feedback about barriers and challenges related to hypoglycemic prevention and the proposed informatics alert. Some of the themes that emerged were: (1) iatrogenic hypoglycemia is not serious enough to justify an alert, (2) patient factors that cannot be controlled (eg, snacking without knowledge of the treating team or resistance to taking recommended insulin doses), (3) provider factors, including knowledge gaps, and, most importantly, (4) concerns regarding alert fatigue. Although the survey respondents almost unanimously agreed that preventing hypoglycemia was an important problem, some of the focus group participants seemed to question its significance. For example, one participant commented as follows:

Many participants voiced concern that the alert system must be structured to prevent alert fatigue. Participants noted that other alert systems were constantly popping up inappropriately leading to people clicking through them and interrupting the workflow.

Overall, the findings from this study indicate that iatrogenic hypoglycemia was perceived as an important problem that could potentially be addressed with a real-time informatics alert. We identified several key design requirements and preferences for the alert: (1) sufficiently accurate, (2) recommendations for care, (3) nondisruptive, and (4) communicated in near real-time to the responsible clinician. Several EMR tools, including an alert indicator embedded in the patient header, glucose management report, and flagged laboratory results, were deemed acceptable formats for consideration of hypoglycemia alerts. Although clinicians indicated that pop-up (interruptive) BPAs would be considered unacceptable because of alert fatigue and disruption of clinical workflow, we were surprised by the acceptability of the use of the hospital’s text messaging system as a channel of communication for the alert because real-time text messaging could be interrupted.

To our knowledge, there are no previous studies soliciting stakeholder input regarding inpatient hypoglycemia informatics alerts. A systematic review and meta-analysis of computerized CDS systems (CDSS) found that push notifications, the ability to execute action, and an interruptive element are the most prevalent features of CDSS that have been evaluated in controlled clinical trials [14]. Absolute incremental improvements in clinical processes or outcomes were demonstrated for CDSS that offered the ability to execute the desired action, and a trend toward improvement was shown for those that allowed for acknowledgment, were behavior-targeted, embedded in the EMR, and interrupted [14].

We recently developed a machine learning algorithm using EMR data that would be considered sufficiently accurate (sensitivity and specificity of approximately 80% and c-statistic of 0.90) based on the input received from stakeholders in this study [22]. As we progress toward translating our machine learning model into a real-time alert, we will seek to consider the key findings from this study to optimize end user buy-in and adoption. Specifically, we intend to develop a near real-time alert that provides specific and actionable recommendations to the ordering provider without disrupting their clinical workflow. The top candidate CDS formats we will consider based on the results of this study are a nonintrusive BPA embedded in the patient header, laboratory result, or existing glucose management report. The selection of the final CDS format will depend in part on our institutional requirements and programing feasibility.

It is important to note that there have been three key system changes at our institution since we completed this study, which may have affected our findings. First, our text messaging system (CORUS) has been replaced by an alternative text messaging platform that resides within the EMR and is accessible through mobile devices (SecureChat). The main difference in the text messaging platform change is that the communication resides within the EMR and is accessible via a mobile app. Communications delivered via SecureChat would allow the user to quickly access other relevant information within the patient’s chart without the need to access an external system. Second, a new feature called Storyboard was released in an Epic software upgrade, which shifted the patient header information from the top of the chart to the left side of the screen. The Storyboard includes patient identifiers and key information, including allergies, infection or isolation, and best practice advisories. At our institution, the Storyboard is already being used to summarize active alerts for a given patient (eg, sepsis) and would be a potential consideration for the development of hypoglycemia alerts. We believe that the inferences of this study related to the patient header would apply to the new Storyboard format. A third system change at our institution has been made in which all services are required to either list an individual or a service pager name (for surgical services) in the first call field. It is expected that the first call field will serve as the source of truth in identifying the responsible provider for a given patient at any given time.

Alert fatigue is a commonly cited limitation in CDSS [28-30]. Despite the fact that avoidance of alert fatigue emerged as one of the most important themes in our analysis, some incongruencies were identified among participants in this study. On the one hand, clinicians strongly opposed a pop-up style BPA; on the other hand, real-time alerting via text messaging was identified as a preferred channel of communication regarding hypoglycemia risk. Although a text message can perhaps be more easily ignored than a pop-up BPA, both forms of alerting could distract the clinician from their current clinical work. On the basis of the overall input we received from our stakeholders, we place greater value on avoidance of alert fatigue and disruption in clinical workflow than immediate notification of the risk of hypoglycemia. Therefore, we will explore several strategies to minimize alert fatigue, such as communicating increased risk in a nonintrusive fashion in the patient header, laboratory result section, or glucose management report.

Future research will assemble an informatics design team to develop prototypes using the most desired formats proposed by stakeholders in this study, and to test these prototypes in a pilot observational study before being widely implemented and evaluated for effectiveness. Ensuring that the alert system follows the CDS Five Rights and adheres and aligns with stakeholder preferences from this study will hopefully improve the usability, adherence, and efficacy of the hypoglycemic alert.