This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on http://humanfactors.jmir.org, as well as this copyright and license information must be included.
Anesthesia information management systems (AIMSs) automatically import real-time vital signs from physiological monitors to anesthetic records, replacing part of anesthetists’ traditional manual record keeping. However, only a handful of studies have examined the effects of AIMSs on anesthetists’ monitoring performance.
This study aimed to compare the effects of AIMS use and manual record keeping on anesthetists’ monitoring performance, using a full-scale high-fidelity simulation.
This simulation study was a randomized controlled trial with a parallel group design that compared the effects of two record-keeping methods (AIMS vs manual) on anesthetists’ monitoring performance. Twenty anesthetists at a tertiary hospital in Hong Kong were randomly assigned to either the AIMS or manual condition, and they participated in a 45-minute scenario in a high-fidelity simulation environment. Participants took over a case involving general anesthesia for below-knee amputation surgery and performed record keeping. The three primary outcomes were participants’ (1) vigilance detection accuracy (%), (2) situation awareness accuracy (%), and (3) subjective mental workload (0-100).
With regard to the primary outcomes, there was no significant difference in participants’ vigilance detection accuracy (AIMS, 56.7% vs manual, 56.7%;
Our findings suggest that it is promising for AIMS use to become a mainstay of anesthesia record keeping. AIMSs are effective in reducing anesthetists’ workload and improving the quality of their anesthetic record keeping, without compromising vigilance.
An anesthesia information management system (AIMS) is a computer-based system that automatically imports real-time vital signs from physiological monitors to replace traditional handwritten records [
Vigilance is the ability to maintain sustained attention over a long period of monitoring [
Situation awareness refers to one’s mental representation of the status of a dynamically changing environment. Situation awareness is measured at the following three levels: perception (level 1), comprehension (level 2), and projection (level 3) [
An AIMS would change the role of anesthetists from active processers of information to passive recipients [
A parallel group experimental design was employed in this study. Ethical approval was obtained from Tuen Mun Hospital (TMH) (NTWC/CREC/17065) and Lingnan University (EC-063/1617). Written informed consent was obtained from all participants in advance and their data were deidentified.
Participants were recruited from among the members of the Anaesthesia and Intensive Care Unit, TMH between September 2017 and March 2018. Participants were eligible if they were resident trainees or specialists. Based on the limited availability of anesthetists, we included 10 participants in each of the two conditions (ie, AIMS and manual), with a total of 20 participants. To achieve simple randomization of group assignment, one experimenter (MKT) placed 10 red (representing the AIMS condition) and 10 green (representing the manual condition) stickers into an opaque envelope and then randomly drew a sticker to generate the allocation sequence. As soon as participants enrolled in the study, they were assigned to a condition according to the allocation sequence.
A full-scale high-fidelity simulation was carried out in an OR at TMH. A clinical scenario specific for this study was designed by three anesthetists (THC, CPC, and KML). The scenario was designed to simulate uneventful monitoring with few critical incidents at intervals [
Apart from the participant, the simulation involved seven people, each with a specific role as follows: (1) senior anesthetist (THC); (2) runner nurse (a registered nurse colleague at TMH); (3) surgeon (CWL); (4) scrub nurse (KML); (5) patient simulator operator (CPC); and (6) two experimenters (MKT and SYWL). The confederates and the patient simulator operator were clinicians from TMH. The two experimenters were researchers from Lingnan University.
Each simulation session was recorded by two digital video recorders; one captured a general view of the OR (
Video capture from the perspective of the operating room (A) and participant (B) while the participant was entering data into the anesthesia information management system during the simulation scenario.
Before the simulation began, participants were given a briefing to introduce them to the purpose of the study. The participants were then informed about the role of each confederate and the function of the patient simulator. In a training session, participants were given instructions and demonstrations on how to respond to assessments of vigilance, situation awareness, and mental workload during the simulation. Participants in the manual condition were also trained on how to manually complete an anesthetic record, because resident anesthetists at the hospital use an AIMS in their usual work practice. The simulation began when the senior anesthetist completed the handover to the participant. The participants were debriefed when the simulation was completed.
The situation present assessment method (SPAM) [
The nine situation awareness queries used in the scenario with their locations of information and their target answers.
Phase, Situation awareness queries | Location of the information | Target answer | |
|
|||
|
Level 1: What is the level of hemoglobin of the patient? | Preoperative assessment | Approximately 11 |
|
Level 2: What is the most possible cause for the patient’s hypertension? |
Physiological monitor (BPa, baseline BP) Understanding of the surgical procedure Medical knowledge |
Tourniquet pain |
|
Level 3: If you do not provide any intervention, what would happen to the BP? |
Physiological monitor (BP, baseline BP) Understanding of the surgical procedure Medical knowledge |
Increase |
|
|||
|
Level 1: What is the patient’s baseline BP? |
AIMS/manual record Physiological monitor |
125/80 |
|
Level 2: What is the most likely cause of the patient’s hypotension? |
Physiological monitor (HRb, BP) Understanding of the surgical procedure Medical knowledge |
Bleeding/volume loss |
|
Level 3: If you do not provide any intervention, what would happen to the end-tidal CO2? |
Ventilator (CO2, baseline CO2, medical knowledge) Understanding of the surgical procedure |
Increase |
|
|||
|
Level 1: How much blood has the patient lost? |
Suction bottle (volume of blood) Communication with nurses (volume of saline drip applied) Blood gauze |
500-700 mL |
|
Level 2: Is the bleeding controlled? Why? |
Suction tubing sound Suction bottle Physiological monitor (BP, HR) Surgical field (eg, blood gauze) |
Yes, there is no more blood in suction tubing/HR and BP become normal |
|
Level 3: If you do not provide any intervention, what would happen to the hemoglobin level? |
Medical knowledge Understanding of the surgical procedure Blood analysis |
Increase. Not enough volume replacement, making the haemoglobin concentration higher. Or decrease. Due to severe blood loss |
aBP: blood pressure.
bHR: heart rate.
There were three primary outcomes as follows: (1) accuracy of detecting suction tubing sounds (ie, vigilance detection accuracy), which were sounds made from actual suction tubing controlled by the scrub nurse (KML); (2) accuracy of correctly answering scenario-specific situation awareness queries (ie, situation awareness accuracy); and (3) self-reported mental workload ratings on The National Aeronautics and Space Administration Task Load Index (NASA-TLX) [
Design of the predetermined vital signs used in the clinical scenario of the simulation and the timeline of vigilance (V), situation awareness (S), and mental workload (W) assessments. BP: blood pressure; HR: heart rate.
The secondary outcomes involved the distribution of the participants’ time across different task activities (ie, task time distribution), the quality of their anesthesia record (ie, anesthesia record completeness), and their attitude toward the AIMS. We assessed participants’ attitude toward the AIMS in terms of trust and acceptance, using a 45-item questionnaire (
Vigilance was operationalized as detection accuracy for each participant. The score was calculated as the proportion (%) of the six tubing sounds that a participant detected. Situation awareness was operationalized as response accuracy, which was calculated as the proportion (%) of the nine situation awareness queries that the participant answered correctly. Each participant’s answers to the situation awareness queries were first evaluated against a predetermined marking scheme. When an answer did not match the target answer, an anesthetist (THC), who was blinded to the condition allocation, helped determine the accuracy of the answer according to expert judgement.
We performed the subjective mental workload measurement at the end of each simulation phase, in which participants rated each NASA-TLX dimension on a scale from 0 (lowest) to 100 (highest). The NASA-TLX comprises six dimensions (mental demand, physical demand, temporal demand, effort, frustration, and performance). The mean overall TLX score for each participant was calculated across the three simulation phases.
Participants’ task activities in the simulation were video recorded and were reviewed to extract data on the different task activities. Task time distribution for each individual task category was computed as a percentage of the time spent on that category over the total time for all four tasks, including (1) entering record data, (2) monitoring the patient (eg, looking at the patient record, physiological monitor, anesthetic gas machine, or simulated patient), (3) performing patient care activities (eg, administering medication into patient’s intravenous access), and (4) interacting with the surgical team (eg, talking to the surgeon, asking the runner nurse to order medication, etc). Data were not coded for tasks that did not fall into any of the four task categories (eg, tidying up equipment wires, walking around the OR, etc).
Two raters assessed the participants’ anesthetic records for completeness using the 15-item checklist by Edwards et al [
The trust and acceptance questionnaire had the following two parts: “trust in the AIMS” (adapted from a scale on trust in automated systems [
Prior to analysis, the Shapiro-Wilk test and Levene test were performed to assess the normality and homogeneity of variance, respectively, of the studentized residuals of the data. The independent sample
According to the directions of the hypotheses, one-tailed significance tests were performed for the primary outcomes, whereas two-tailed tests were performed for the secondary outcomes. Task time distributions of the four tasks were compared between the two conditions with Bonferroni correction to obtain a more stringent alpha level of .0125 (.05/4).
All 20 participants completed the trials without any dropout (
CONSORT disgram for the simulation study.
There was no significant difference in vigilance accuracy between the AIMS (mean 56.7%, SD 32.6%) and manual conditions (mean 56.7%, SD 31.6%) (t18=0.00,
Some video data were not coded (30% in the AIMS condition and 26% in the manual condition), as they either could not be classified or involved tasks that did not fall into our predefined task categories. Of the data that were coded according to the four task categories, only the proportion of time spent on record data entry differed significantly between the AIMS (mean 26.0%, SD 4.9%) and manual conditions (mean 33.7%, SD 6.9%) (t18=−2.87,
Despite the increasing adoption rate of AIMSs in hospitals [
AIMS use might have two advantages over manual record keeping with respect to mental workload. First, the lower subjective mental workload with AIMS use might be a product of reduced physical movements. Informal inspection of our GoPro video data revealed that participants in the manual condition exhibited extensive head movements owing to the shifting of attention between the physiological monitor and the paper anesthesia chart. These movements may imply that more cognitive and perceptual activities (eg, remembering, looking, and searching for information) are involved in manual record keeping, and thereby, they result in higher subjective mental workload. Second, manual record keeping might have placed a high demand on participants’ prospective memory (remembering a future task) [
The secondary outcomes indicated further benefits of AIMS use. First, participants who used the AIMS spent about 8 percentage points less of their total time on record data entry than those who used manual record keeping. This result confirms previous findings that electronic record keeping allows anesthesia residents to spend less time on record keeping as compared to that with manual record keeping [
Compared with previous studies on AIMS use that only examined visual vigilance [
This study had six limitations. First, our simulated scenario only represented anesthetic cases that involve an uneventful period followed by critical incidents. Therefore, our findings can only be applied to the context of anesthesia with critical incidents. In anesthesia, many cases occur without any critical events. When the anesthetic procedure is uneventful, the effect of AIMS use on anesthetists’ vigilance and situation awareness might be different because complacency might arise, and this warrants further investigation.
Second, our participants were more accustomed to AIMS use than manual record keeping in their usual practice because junior anesthetists at TMH are trained on the AIMS but not on manual record keeping. Therefore, participants in our simulation had to be retrained on manual record keeping for comparison. While this retraining might seem artificial, it was the aim of TMH’s Department of Anaesthesia & ICU to investigate the tacit assumption of the effectiveness of AIMS use over manual record keeping. Retraining in the manual condition might have increased participants’ perceived mental workload, degraded their vigilance, and decreased their record keeping efficiency. This possible confounding factor could be addressed in future studies by sample screening or providing participants with prolonged training in manual record keeping.
Third, the findings of our study cannot be generalized to all models or brands of AIMSs. Other models of AIMSs might have different functions or interfaces and might interact with anesthetists differently.
Fourth, the participants, experimenters, and confederates were not blinded to the condition assigned to each participant owing to the nature of the manual and automated record keeping conditions.
Fifth, although our results suggest that AIMS use reduced the time spent on record data entry, it is unclear whether the time reduction led to an increase in time spent on monitoring patients or performing patient care activities. This could be addressed in future studies by examining how anesthetists reallocate the time saved with AIMS use to other tasks.
Sixth, we used a GoPro camera attached to each participant’s head in an attempt to capture visual data. However, the GoPro camera, at its best, could only provide us with the participant’s gaze direction. If accurate visual attention data are to be gathered, a mobile eye tracker should be used in future studies. Eye tracking data would allow for not only better inference of participants’ visual attention in general, but also identification of what activities they focus on when not interacting with the AIMS.
Despite the increasing popularity of AIMSs in hospitals, no previous studies have analyzed their effects on comprehensive monitoring performance. The findings of this study provide support for the adoption of AIMSs in the OR by demonstrating a number of benefits of AIMS use, including reducing anesthetists’ perceived mental workload, saving their time spent on data entry, and producing complete anesthetic records, without compromising vigilance. Moreover, the majority of our anesthetists expressed a positive attitude toward trusting and accepting AIMSs in the OR.
The level of automation in health care is likely to increase as medical technology advances. It is important to know the effects that automation will have on patient care, as it could affect clinicians’ care quality and, ultimately, patients’ well-being and safety.
The goal-directed task analysis.
Pooled situation awareness requirements for the scenario.
Questionnaire for trust and acceptance of anesthesia clinical information systems.
CONSORT-eHEALTH checklist (V 1.6.1).
anesthesia information management system
goal-directed task analysis
National Aeronautics and Space Administration’s Task Load Index
operating room
situation present assessment method
Tuen Mun Hospital
This study would not have been possible without support from the Department of Anaesthesia and Intensive Care and the Quality and Safety Division at Tuen Mun Hospital. We would like to sincerely thank Tuen Mun Hospital’s anesthetists who participated in the study and Francis Leung Wai Sing who generously made time to prepare and participate in the simulation. We would also like to express our gratitude to Professor Penelope Sanderson and Professor Robert Loeb for their encouragement and valuable comments that helped us improve the research. This research was supported by a postgraduate studentship from Lingnan University awarded to MKT.
None declared.