This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on http://humanfactors.jmir.org, as well as this copyright and license information must be included.
Current anesthesia workspaces consist of several technical devices, such as patient monitors, anesthesia machines, among others. Commonly, they are produced by different manufacturers; thus, they differ in terms of their modus operandi, user interface, and representation of alarms. Merging the information from these devices using a single joint protocol and displaying it in a single graphical user interface could lead to a general improvement in perioperative management. For this purpose, the recently approved and published Institute of Electrical and Electronics Engineers 11073 service-oriented device connectivity standard was implemented.
This paper aims to develop and then evaluate an anesthesia workstation (ANWS) functional model in terms of usability, fulfillment of clinical requirements, and expected improvements in patient safety.
To compare the self-developed ANWS with the conventional system, a pilot observational study was conducted at the University Hospital Aachen, Germany. A total of 5 anesthesiologists were asked to perform different tasks using the ANWS and then the conventional setup. For evaluation purposes, response times were measured and an interaction-centered usability test with an eye-tracking system was carried out. Finally, the subjects were asked to fill in a questionnaire in order to measure user satisfaction.
Response times were significantly higher when using the ANWS, but decreased considerably after one repetition. Furthermore, usability was rated as excellent (≥95) according to the System Usability Scale score, and the majority of clinical requirements were met.
In general, the results were highly encouraging, considering that the ANWS was only a functional model, as well as the lack of training of the participants. However, further studies are necessary to improve the universal user interface and the interplay of the various networked devices.
In recent decades, the development of new innovative medical devices has led to significant improvements in patient safety, quality of supply, and economic advantages. Conventional anesthesia workspaces, for instance, consist of many different technical devices, including (1) a patient monitor, which records and displays the patient's vital parameters, such as electrocardiogram, body temperature, blood pressure, oxygen saturation, pulse rate, and respiratory rate; (2) an anesthesia machine, which is used to support the administration of anesthesia and for mechanical ventilation; and (3) syringe pumps, which administer intravenous anesthesia. Nowadays, all of these devices are the basis for standard patient care in anesthesiology [
However, medical devices assembled for clinical applications are usually produced by different manufacturers. They vary in terms of their modus operandi, user interface, and representation of alarms [
In order to solve the issues discussed, an interdisciplinary consortium composed of engineers, computer scientists, and physicians from approximately 50 German organizations (ie, research institutes, hospitals and clinics, and medical companies) initiated the Secure Dynamic Networking in the Operating Room and Clinic (OR.NET) research project, which was funded by the German Federal Ministry of Education and Research (grant number 16KT1238). Two of the main goals were to develop a single and sophisticated protocol—the service-oriented device connectivity (SDC) family of standards, formerly the Open Surgical Communication Protocol—for medical device communication, as well as a new anesthesia workstation (ANWS), focusing at the same time on human-computer interaction (HCI) and safety concepts [
In addition to technical interoperability issues, such as definition of data and communication protocols, safety and risk management is also part of the 11073-SDC communication protocol extensions 10207, 20702, and 20701. They were approved and published by the IEEE in January 2019. The so-called
The ANWS functional model was developed to demonstrate and examine the capabilities of medical device interconnections based on the SDC standards; the model is represented in
The objective of this paper is to evaluate the ANWS functional model developed during the OR.NET research project in terms of usability, fulfillment of clinical requirements, and expected improvements in patient safety.
The anesthesia workstation (ANWS) functional model. The interface screen consists of six main elements: (a) patient context and alarms; (b) device panel: overview of all devices connected to the ANWS; (c) view mode selector; (d) area reserved for the selected view; (e) workflow process; and (f) eye-tracking system.
In order to evaluate the usability of the OR.NET ANWS, and consequently the HCI, a study involving 5 professional volunteers was conducted at the University Hospital Aachen, Germany. Potential benefits of the functional model over a conventional system were examined. Therefore, every participant started with the OR.NET setting, followed by the conventional setup a few weeks later. During a total hip arthroplasty (ie, total hip replacement) surgery simulation, the usability of both systems was evaluated by measuring the usability criteria (ie, effectiveness, efficiency, learnability, and user satisfaction); criteria were measured by (1) using the response time and (2) applying the think-aloud method. Thinking aloud is often used in usability testing by asking the participants to say whatever comes into their mind while performing certain tasks. This method enables insights into the participant’s cognitive processes, including perception, doing, and feeling [
The ANWS was installed on a desktop computer with the following properties: Quad-Core i5 CPU, 8 GB RAM, 250 GB solid-state drive, and a Radeon R7 200 dedicated graphics card with 1 GB graphics double-data rate type 5 memory. It was started as a stand-alone executable based on the .NET Framework 4.6 and OpenStackClient C# library (OSCLib C#), version 0.97_09 (SurgiTAIX AG).
An OR.NET-SDC device simulator (Ilara GmbH) was used during the study to simulate measurements and output data for the following medical devices: syringe pump, patient monitor, and anesthesia machine. Thus, predetermined clinical scenarios could be reproduced. The simulator transmitted scheduled numerical data (eg, heart rate, blood pressure, and oxygen saturation) as well as physiological waveforms (eg, electrocardiography and respiratory curves) inside the network. In addition, it was able to respond to external commands; these were used to set parameters in the simulated devices (eg, infusion rates of syringe pumps). It was started as a stand-alone executable based on Microsoft Office 2016, .NET Framework 4.5, and OSCLib C#, version 0.96_00. The ANWS and the simulator were connected using the SDC through an internal network on the test computer.
To monitor the users’ behavior (ie, the users’ visual attention on user interface elements), we used an eye-tracking camera system—Gazepoint GP3 HD with Gazepoint Analysis and Gazepoint Control recording software, version 3.5 (Gazept). This was installed below the external monitor, as displayed in
The subjects were asked to report or express every single thought while solving the tasks (ie, think-aloud method). During the study, an audio record was created for future analysis. Here, the microphone from the LIVE! Cam Chat HD webcam (Creative) was used.
In the first phase, the participants had to carry out 42 tasks. As described in
At the end of the study, each participant filled in a survey that included questions about sociodemographic aspects, technical expertise, motivation, study conditions, and features tested. For each tested feature of the ANWS (see
Tasks carried out by the subjects in both study phases.
Task | Study phase | Number of repetitions |
Select type of surgery and load respective workflow | ANWSa | 1 |
Customize workstation window | ANWS | 1 |
Update workflow process | ANWS | 4 |
Read blood pressure | ANWS, Cb (PMc) | 4 |
Read temperature | ANWS, C (PM) | 3 |
Read oxygen saturation | ANWS, C (PM) | 7 |
Read heart rate | ANWS, C (PM) | 2 |
Read airway pressure | ANWS, C (AMd) | 4 |
Read fraction of inspired oxygen | ANWS, C (AM) | 1 |
Read respiratory minute volume | ANWS, C (AM) | 1 |
Read respiratory compliance | ANWS, C (AM) | 1 |
Set positive end-expiratory pressure (ie, respirator parameter) | ANWS, C (AM) | 2 |
Set infusion flow rate | ANWS, C (SPe) | 4 |
Fetch and complete Surgical Safety Checklist | ANWS, C | 3 |
Consider a pulmonary embolism | ANWS, C | 1 |
Switch workspace view | ANWS | 2 |
Check for intelligent alarms | ANWS | 1 |
Total tasks | ANWS, C | 42 (ANWS); 33 (C) |
aANWS: anesthesia workstation.
bC: conventional setting.
cPM: patient monitor.
dAM: anesthesia machine.
eSP: syringe pump.
Overview of all features integrated into the anesthesia workstation and respective descriptions.
Featurea | Tested | Description |
Automatic documentation | No | Possibility of saving all information provided by a service-oriented device connectivity-compatible device inside the OR.NETb network, including meta information, through the ANWSc. |
Compilation of all alarms | No | Alarms provided by any anesthesia-related device (eg, syringe pump, anesthesia machine, and patient monitor) are collected and displayed in the ANWS. |
Control devices from other departments | No | Bidirectional control of devices (eg, anesthesiologists can control surgery devices and surgeons can control anesthesia equipment). |
Display content from other departments | No | Capability to integrate several or even single measures or parameters in any connected device (eg, to display the current blood pressure, measured by the patient monitor, in the surgical microscope). |
Cross-device interaction | Yes | Enables (eg, control of) diverse surgical devices using a single universal footswitch, button, or joystick. |
Decision-support system | Yes | Context-adaptive hints and suggestions are displayed based on the currently ongoing surgical intervention, the actual workflow step, eventual patient-related problems, etc. |
Segregated alarms | No | All alarms are classified as medical or device-associated (ie, technical) alarms. |
Surgical Safety Checklist | Yes | Display of the Surgical Safety Checklist as an integrative part of the surgical workflow. |
Unified surface | Yes | Fusion of information from several devices in a single graphical user interface. |
Workflow management | Yes | Generation from a workflow based on previous data (ie, database). After each step, the process is updated automatically or manually, enabling significantly improved (ie, predictive) planning. |
aAll features were subjectively rated; five of those were additionally included in a user test scenario.
bOR.NET: Secure Dynamic Networking in the Operating Room and Clinic.
cANWS: anesthesia workstation.
System Usability Scale (SUS) scores. Graphical overview of the adjective ratings, acceptability scores, and school grading scales, in relation to the average SUS score. Figure adapted from Bangor et al [
Finally, participants were asked to prioritize the tested features according to the relevance in their daily work. Furthermore, they were requested to compare the OR.NET ANWS with common or conventional systems in terms of personal preference and impact on patient safety.
In the conventional phase, the subjects were asked to carry out 33 tasks (see
In this evaluation, participants used a Julian anesthesia machine (Draeger Medical), a Perfusor Space syringe pump (B Braun), a Datex-Ohmeda AS patient monitor (General Electric), and a filing folder including the printed SSCs for the three operation phases: (1)
All 5 subjects—aged 30-42 years; 3 females (60%) and 2 males (40%)—participated in this study voluntarily. The group was composed of anesthesiologists—2 assistant physicians (40%) and 3 consultant anesthesiologists (60%)—from the Department of Anesthesiology of the University Hospital Aachen. The group was heterogeneous regarding practical experience: 3 physicians (60%) had more than 5 years of experience; 1 (20%) had 3-5 years of experience; and 1 (20%) had 1-2 years of experience. None of them had any knowledge of the study scenario or tasks. Only 1 (20%) of the participants had seen the GUI design of the ANWS before, without any connection to the study scenario or tasks. According to the guidelines of the ethics committee, no formal approval was necessary to conduct the study.
Illustration of the study setup: (a) patient monitor, (b) anesthesia machine, (c) syringe pump, and (d) patient simulator.
The primary outcome parameters in this study were the response times and the capability of the subjects in successfully solving each task. As previously discussed, eye tracking was used in this study. The first aim was to identify elements that distracted attention from more important information, known as “vampire effect.” The second aim was to find elements with a hidden affordance. Lastly, eye-tracking information was essential to find areas of interest as well as hitherto unknown impacts or suggestions for improvements.
Using the think-aloud method, it was possible to achieve a better understanding of the participant’s line of thought, reasons for the occurrence of problems or misunderstandings, if applicable, and to identify potential further improvements to the system.
The questionnaire, in turn, was used to track background knowledge and motivation. Information on technical know-how and work experience was requested in order to determine the actual impact of each participant on the study, including a possible halo effect (ie, form of cognitive bias arising from an overgeneralization, such as a limited amount of evidence, the influence of preconceived beliefs, or a priori hypotheses; it is particularly prone to occur among participants who are enthusiastic about technology) [
Statistical analysis was performed using SPSS Statistics for Windows, version 22.0 (IBM Corp). Normal distribution was analyzed with Kolmogorov-Smirnov and Shapiro-Wilk tests. When detecting significance in the comparison of means (ie, between the conventional and OR.NET settings), the Mann-Whitney
First part of the questionnaire.
Questions and responses | Scorea, median (min, max) | |
I often use technical innovation in everyday life. | 4 (4, 4) | |
I’m skeptical of new technologies. | 1 (1, 2) | |
I often use technical innovations to make my life easier. | 4 (3, 4) | |
I quickly get used to using new technologies. | 4 (3, 4) | |
I can easily use new technologies. | 4 (3, 4) | |
I don’t like surgical robots in medicine. | 1 (1, 2) | |
In general, more technology should be used. | 4 (2, 4) | |
New technologies endanger society. | 1 (1, 2) | |
A smartphone (private) | 4 (4, 4) | |
A smartphone (work) | 4 (4, 4) | |
A tablet (private) | 3 (2, 4) | |
A tablet (work) | 3 (2, 4) | |
A notebook or PC (private) | 4 (1, 4) | |
A notebook or PC (work) | 4 (1, 4) | |
Another device; private (eg, smart glasses or smart watch) | 1 (1, 4) | |
Another device; work (eg, smart glasses or smart watch) | 2 (1, 3) | |
Scientific studies do not support health care delivery and outcomes. | 1 (1, 1) | |
I expect a compensation or reimbursement for expenses. | 1 (1, 3) | |
Personally, I do not find questionnaires useful for gathering information on individual or collective perspectives. | 1 (1, 3) | |
I participate in studies to learn. | 3 (2, 4) | |
I felt great pressure to perform. | 1 (1, 3) | |
I felt excessively challenged. | 1 (1, 1) | |
I was stressed. | 2 (1, 2) | |
My motivation was high. | 4 (3, 4) | |
I was focused. | 4 (3, 4) |
aScores were as follows: 0 (
All participants (5/5, 100%) agreed that such studies might improve health care delivery and outcomes. In fact, most of the volunteers wanted to participate in order to learn more (4/5, 80%) and did not expect any compensation or reimbursement for expenses (4/5, 80%). In addition, they considered the questionnaires to be helpful for gathering information on individual and collective perspectives.
Regarding the study itself, the participants did not feel pressured, stressed, or challenged. Instead, they were highly motivated and focused in trying to solve the tasks. In fact, 2 physicians (40%) had gained experience in telemedicine through the emergency medical service in Aachen, Germany [
One of the first tasks in this study consisted of finding the correct SOP. In the ANWS setting, the SOP was automatically generated by the decision-support engine with no delay and was embedded in the workflow management. However, in the conventional setting, the volunteers required 26.8 seconds to accomplish the task.
As described in
When the subjects were advised to set the PEEP in the anesthesia machine and in the workstation for the conventional and ANWS setting, respectively, the same phenomenon was observed (ie, they required less time when using the conventional approach: conventional, 2 seconds [IQR 2-3]; ANWS, 3 seconds [IQR 3-4];
As represented in
During the implantation of prosthetic sockets in total hip arthroplasties and in surgeries in general, several complications may occur. Within the surgery simulation, a pulmonary embolism was mimicked (see
All areas of interest were identified by each participant. However, the usability study demonstrated that improvements in some elements of the GUI were still necessary. When starting the surgery workflow, the button used to select the patient and the corresponding operation, in this case a total hip arthroplasty, showed a clear case of hidden affordance. Furthermore, it was not intuitively obvious which button needed to be clicked to update the workflow process. Although eye movements indicated searching and the participants often looked directly at the correct button, they had difficulties in understanding its purpose. Another significant case of hidden affordance was recognized when dragging and dropping the device panels from the device selector (see
Tukey’s box plots comparing the response times of both groups: anesthesia workstation (ANWS; left) versus conventional setting (right). The “o” represents for outliers, “x” marks the arithmetic mean, and “*” indicates statistical significance (
The second part of the questionnaire: the System Usability Scale results presented by Tukey’s box plots. Each feature was rated from 0 (
In the questionnaire, the volunteers were also asked to sort ANWS features—tested and untested—according to their preference, from 1 (
At the end of the questionnaire, the subjects were asked to choose, for each criterion, their favorite system: ANWS or conventional. The results, represented in
Better monitor the parameters of different devices.
Always complete the SSC.
Get important additional information.
Improve the overall usability of the connected devices.
Get a better overview of the current surgical step.
Increase patient safety.
In general, the subjects demonstrated clear opinions with the exception of subject 2; he had a neutral opinion with regard to item 4. Although the majority of the subjects considered the ANWS functional model as their personal favorite system, 2 out of 5 (40%) considered the use of the ANWS to be more complicated when rating the
Tested (top) and untested (bottom) features were rated by the subjects from 1 (
Graphical representation of the subjects’ favorite system, anesthesia workstation or conventional systems, according to seven criteria.
In this study, 5 anesthesiologists were requested to test and evaluate a newly self-developed ANWS functional model. To test its usability, the subjects were requested to perform several tasks using the ANWS and the conventional system. The ANWS was deemed noninferior compared to the well-known system that was used daily.
All subjects considered themselves
To analyze the learning curve, some tasks were repeated. As expected, during the first try, the subjects requested more time (2.5 seconds [IQR 0.75-4.25] vs 1 second [IQR 0-3]) when using the ANWS. During further repetitions, the response time decreased significantly, resulting in a time span of less than 1 second in each setting. As demonstrated in
Unfortunately, hospitals frequently utilize medical devices from various manufacturers, even for the same appliance classes, such as patient monitoring, ventilators, and syringe pumps, which means different modus operandi. Consequently, the physician must be capable of working with all of them and of changing over without any time delay. As a result,
Another important advantage of the ANWS is the fact that the World Health Organization SSC is integrated into the surgical workflow panel so that it must be filled in. Therefore, this feature contributes to improved patient safety and quality of care since SSCs are often neglected in clinical practice. In addition, the functional model also includes a decision-support system aimed at assisting the medical team in the decision-making process. It uses patient data (eg, vital parameters) as input measures and combines them with mathematical models and algorithms. When potential abnormal changes are detected, the physicians are informed (eg, by an alarm). As demonstrated in the Response Time section of the Results, the anesthesiologists required more time to detect the pulmonary embolism when using the conventional system. In contrast, when using the ANWS, the subjects were promptly informed about this complication and could start with the treatment directly. Of course, the decision-support system does not replace the physician, but is able to support him or her and, therefore, improve the quality of care.
Regarding the questionnaire, the subjects considered automatic documentation a very important feature since it contributes to a reduction in the workload. Concerning the remaining features, no consensus was reached because preferences are usually subjective and vary from individual to individual. However, in general, a great acceptance of the ANWS functional model was observed. Indeed, all tested features were graded as at least
Despite the positive feedback, there is still room for improvement. Taking the eye-tracking and think-aloud method analyses into account, two controls indicated hidden affordance: the workflow management controls and the drag-and-drop mechanism of the device panels need to be adapted to improve usability. Fortunately, no “vampire effects” were detected and all areas of interest were identified clearly.
This study has certain limitations, which need to be addressed. First, due to the early stage of development of the ANWS, this pilot study was carried out with only 5 participants, which is the minimum requirement for formative usability tests within the usability engineering process for medical devices, according to the International Electrotechnical Commission (IEC) 62366 standard. Therefore, the statistical capacity is still limited for this pilot study.
Second, not all features of the functional model were investigated (eg, automatic documentation). Third, the experimental settings were not very close to reality, especially when considering the ANWS testing. Due to the lack of SDC-compatible devices, all measures were simulated digitally but were not derived from a patient or a patient simulator. Finally, the functional model is not certified as medical device, hence, a simulator instead of
Despite these drawbacks, this study is, to the best of our knowledge, the first analysis regarding the acceptance of an innovative and pioneering ANWS, based on a manufacturer-independent communication protocol, namely SDC.
Although the ANWS only turned out to be noninferior compared to the conventional setting, this is already promising. We believe that the ANWS would have been superior if the following had been provided:
Adequate training of the professional users, analogous to the conventional system.
Optimization of the GUI in terms of usability aspects and design.
Integration into a clinical environment including
Actual tasks during real anesthesiologic workflows, such as anesthesia induction, maintenance, and complications.
Complex scenarios that benefit from
Although the technologizing of hospitals, especially in operation theaters, is increasing, medical devices and IT systems mainly work as stand-alone versus networked systems. Consequently, progress in health care digitization is very slow. In this project, we developed an anesthesia dashboard, enabling various sophisticated features based on open device interconnection. In the meantime, our project team contributed to the development and approval of SDC, a manufacturer-independent IEEE 11073 standard for medical device networking, which was approved as a worldwide accepted standard in January 2019.
The way is now paved for manufacturers to equip their medical devices with SDC-compliant interfaces, enabling interconnectivity in the operation theater and elsewhere in health care.
anesthesia workstation
graphical user interface
human-computer interaction
International Electrotechnical Commission
Institute of Electrical and Electronics Engineers
interquartile range
Secure Dynamic Networking in the Operating Room and Clinic
OpenStackClient C# library
positive end-expiratory pressure
service-oriented device connectivity
standard operating procedure
Surgical Safety Checklist
System Usability Scale
The project was supported by the German Federal Ministry of Education and Research. In addition, we would like to thank Uwe Wainwright and Michael Schauwinhold for their support in operating the AIXTRA Patient Simulator.
MO and MC are employed by Docs in Clouds GmbH (Aachen, Germany), providing telemedical services and consulting regarding smart hospital solutions. MC is CEO of Ilara GmbH (Herzogenrath, Germany), a provider of medical software, including OR.NET device simulators and connectors. AJ and MC are members of the executive board of the nonprofit association OR.NET eV.