Prioritizing Human Factors in the Clinical Translation of Mixed Reality Surgical Guidance

Introduction

Mixed reality and augmented reality technologies have emerged as promising tools for intraoperative visualization, surgical navigation, training, and decision support [1]. Their appeal in surgery lies in the ability to present patient-specific or task-relevant information directly within the user’s field of view, with the potential to reduce reliance on external monitors and improve spatial understanding of anatomy [1,2].

Recent reviews have highlighted the potential benefits of mixed reality in the operating room, including gains in efficiency, learning, and anatomical understanding, while also identifying persistent barriers to implementation, such as ergonomic limitations, technical complexity, narrow field of view, battery limitations, and broader usability challenges [3,4]. Reviews focused specifically on optical see-through head-mounted displays in surgery have likewise highlighted recurring problems in perception, ease of use, context adaptation, interaction design, and occlusion [5]. More recent health care–oriented work has emphasized the need for evaluation frameworks that reflect authentic clinical and human visual conditions rather than idealized bench environments [6].

Despite growing technical interest in surgical mixed reality, the human factors conditions required for safe and effective use remain insufficiently addressed in relation to real-world clinical translation [7]. In this paper, the term “human factors” is used in a broad ergonomics sense to refer to the interactions between people, technologies, tasks, environments, teams, and organizational conditions that influence safety, performance, and implementation in practice [8]. This Viewpoint is intended for researchers, developers, clinicians, and human factors specialists involved in the design, evaluation, and clinical translation of mixed reality surgical guidance systems. Against this broader perspective, we focus on 3 domains that, in our view, represent especially important translational priorities for mixed reality surgical guidance: physical characteristics, dexterity and motor demands, and environmental resilience. These are not presented as the only relevant human factors concerns but as recurring practical conditions that must be addressed if mixed reality systems are to move beyond proof-of-concept use and function reliably in routine operative settings. Mixed reality may offer genuine benefits in spatial awareness and in-field information access, but these advantages are likely to translate reliably into practice only if these practical conditions of use are addressed. Broader team, organizational, training, and governance issues remain essential to safe implementation, but they are treated here primarily as part of the wider systems context rather than as the main detailed focus of this Viewpoint.

Human Factors Framework for Mixed Reality Surgical Guidance

A useful way to frame mixed reality surgical guidance is through a systems human factors lens. In this paper, our framework is informed particularly by Moray’s [8] systems approach to ergonomics, which emphasizes that performance and safety are shaped not only by individual users or technologies but also by interacting influences across multiple levels of the wider work system. This orientation is also consistent with broader sociotechnical and health care human factors perspectives that consider how individual, task, team, organizational, and wider system conditions interact in practice [8-10].

From this perspective, performance is shaped not only by the technology itself but also by interactions across multiple layers of the clinical work system, including individual user-device interaction, task and procedural context, team and environmental context, organization, training, and governance [8-10]. This means that safety, usability, effectiveness, and adoption depend not only on registration accuracy or display performance but also on physical ergonomics, visual perception, dexterity and motor control, workflow integration, calibration and verification, communication, supervision, and the ability to recognize failure and recover safely when the system becomes unreliable.

This framing matters because mixed reality surgical guidance is often discussed primarily in technical terms, whereas its real-world performance depends on how it functions within a layered clinical work system. Figure 1 [8] summarizes this systems human factors perspective and illustrates how factors operating at user, task, team, environmental, and organizational levels may influence clinical and system outcomes. It should therefore be understood not only as a conceptual summary but also as a framework for guiding evaluation. In particular, it highlights how constraints at one level, such as user perception or interaction burden, may be shaped by and have consequences for task demands, team communication, environmental conditions, and organizational arrangements.

The next section examines the 3 domains we consider to be most important (physical characteristics, dexterity and motor demands, and environmental resilience), focusing on how they influence whether mixed reality systems can be comfortably worn, reliably interacted with, and used robustly under operative conditions.

‎

Key Human Factors Priorities for Mixed Reality Surgical Guidance

Translational Context for Mixed Reality Surgical Guidance

Published work on mixed reality in surgery spans operative guidance, planning, education, and navigation, with head-mounted devices such as Microsoft HoloLens and Magic Leap featuring prominently [3-5]. Much of this work has centered on navigation-heavy applications, particularly in maxillofacial and neurosurgical settings, where rigid anatomy supports relatively stable registration and validation [5,6]. Many studies remain based on phantom, cadaveric, pilot clinical, or early feasibility designs rather than definitive comparative evaluation [3-5]. From a human factors perspective, attention has focused mainly on user-device interaction, ergonomics, visualization, interface use, and selected workflow issues, while broader team, organizational, training, and governance questions remain less developed. In our view, this imbalance helps explain why mixed reality systems may show promise in early technical studies while remaining incompletely prepared for routine clinical use.

Physical Characteristics as a Translational Priority

Comfort, Weight, and Wearability

Ergonomic burden is one of the most consistently reported physical issues in the mixed reality surgical literature. Magalhães et al [3] identified ergonomic limitations, restricted field of view, and battery autonomy as persistent barriers to operating room adoption. Brockmeyer et al [11] likewise noted that surgeons often rate comfort reasonably well overall but still identify headset weight and size as important shortcomings of current HoloLens-based systems. Similarly, Doughty et al [5] highlighted device obtrusiveness and weight as contributors to both perceptual and ergonomic limitations.

Taken together, these observations suggest that wearability remains a translational issue rather than a minor design inconvenience. Broader evaluation work has also argued that wearability should be assessed under authentic clinical conditions, as laboratory testing may underestimate cumulative strain during prolonged use [6]. This is highly relevant in surgery, where even modest head or cervical load may influence posture, endurance, and willingness to use the system over the duration of a case. In our view, future human factors studies should treat comfort and wearability as core determinants of acceptability rather than as secondary user satisfaction measures.

Field of View and Visual Limitations

Limited field of view is repeatedly described as a key constraint [3,5,12]. Although these platforms may improve information access by placing digital content within the user’s line of sight, the restricted display area can limit how much holographic information can be perceived at once and can reduce the practicality of direct overlay guidance [5]. Feasibility studies summarized in the review literature also note that HoloLens devices have a relatively narrow diagonal field of view, even where users remain broadly satisfied with image quality for targeted tasks [6,13].

From a translational perspective, field of view should not be regarded simply as a hardware specification. It has direct implications for perceptual continuity, attention management, and the extent to which holographic guidance can be integrated into procedural workflow. Future evaluation should therefore examine how field-of-view constraints affect interpretation, task switching, and confidence in guidance during clinically relevant tasks.

Visual Perception and User-Specific Calibration

Physical user differences matter because perception in optical see-through systems is shaped by interpupillary distance, vergence-accommodation conflict, display calibration, and subjective depth interpretation [5,6]. Doughty et al [5] described incorrect interpupillary distance estimation, calibration instability, and depth perception limitations as recurrent contributors to perceptual error in surgical use of optical see-through head-mounted displays. The 2025 augmented reality head-mounted device (AR-HMD) evaluation review emphasized the importance of replicating human visual conditions, including interpupillary distance, pupil size, and observation geometry, when evaluating these devices [6].

These issues are especially important in surgery because a system may appear visually acceptable in generic testing while still introducing subtle perceptual errors that become clinically relevant during alignment, targeting, or spatial judgment. In our view, user-specific calibration should be treated as a core human factors issue rather than a purely technical one, particularly where holographic guidance is expected to support procedural precision.

Fatigue, Dizziness, and Visual Strain

Several reviews note that prolonged headset use may induce headaches, dizziness, motion-related discomfort, or visual fatigue [3,5,11]. The mixed reality operating room review explicitly identified dizziness and reduced field of view among the main ergonomic challenges [3]. These observations suggest that physical user research should extend beyond static comfort ratings and consider cumulative fatigue and tolerance over the duration of a case [3,6].

In a translational context, this matters because surgical mixed reality systems may be acceptable in short feasibility scenarios yet become less tolerable during longer cases or repeated use. Fatigue and visual strain should therefore be treated as key aspects of early evaluation, particularly for systems intended for routine intraoperative use.

Implications

Mixed reality systems cannot assume a one-size-fits-all user model [6,13]. Variation in vision, posture, wearable tolerance, and interaction with eyewear or theater attire is likely to affect safe use yet remains insufficiently addressed in current surgical studies [5,6]. In our view, future evaluation should explicitly incorporate user diversity and test whether systems remain comfortable, interpretable, and usable across realistic ranges of anthropometric and visual variation.

Dexterity and Motor Demands as a Barrier to Usable Interaction

Gesture and Voice Interaction Burden

Gesture and voice are among the most common interaction modes in surgical mixed reality systems [3,4]. The optical see-through review found that gestures and voice were the preferred interaction paradigm in many systems, especially HoloLens-based applications [5]. However, interaction burden remains a recurring problem [4,5]. In feasibility work, voice commands and gesture functions have often been rated less favorably than image quality or perceived overlay utility, indicating that interaction reliability may lag behind visual promise [5].

In our view, this is an important translational warning. Interaction methods may appear acceptable in proof-of-concept testing yet still prove impractical when deployed in time-pressured, sterile, and interruption-prone clinical environments. Human factors evaluation should therefore move beyond asking whether a command can be recognized and instead examine whether interaction remains efficient, intuitive, and low burden in the context of real surgical workflow.

Fine Motor Performance and Latency

Surgical performance depends on precise coordination, and mixed reality systems can interfere with this when interaction design introduces lag, poor input mapping, or repeated corrective actions [4-6]. The AR-HMD evaluation review highlighted that fine motor requirements are particularly critical in health care contexts requiring precision and that latency or poor finger tracking can degrade manipulation reliability [6]. Related work has also suggested that current systems may not yet offer the level of fine motor responsiveness required for delicate clinical use [4,6].

This has clear implications for surgical guidance. In high-precision environments, even minor delays or inconsistencies may alter the user’s willingness to rely on the interface, increase hesitation, or encourage workarounds that bypass the system altogether. We therefore suggest that future evaluation should explicitly test the relationship between latency, manual precision, and operator confidence, rather than treating technical responsiveness as a purely engineering metric.

Hand-Eye Coordination and Perceptual-Motor Coupling

The optical see-through review identified hand-eye coordination as a reported human factors issue in some studies, particularly where users had to align tools, physical targets, and holographic guidance simultaneously [5]. Where interface control required users to adjust overlays, invoke commands, or reinterpret a partly registered scene, this increased both motor and cognitive demand [5].

In our view, this challenge is central to the real-world usability of mixed reality guidance. Surgical systems should not assume that visual overlay automatically improves performance. Instead, future studies should examine whether the system supports or disrupts perceptual-motor coupling during actual task execution, particularly in procedures that rely on fine targeting or continuous manual control.

Ease of Use and Workflow Interruption

Dexterity-related challenges are not limited to movement precision alone; they also include the need to interrupt procedural flow to manipulate content, navigate menus, or repeat commands [5]. Consistent with this, published work to date has described workflow disruption arising from the need to manually adapt virtual content during procedures [5,11]. In a sterile setting, any control modality that competes with operative hand use or requires repeated correction may reduce practical utility, even when the display itself appears technically sound [5,13].

This is where the distinction between technical functionality and clinical usability becomes most apparent. A system may function correctly in a narrow technical sense yet still impose enough interaction burden to make it unattractive or unsafe in practice. We therefore argue that workflow interruption should be treated as a primary human factors outcome in future evaluations of mixed reality surgical guidance systems.

Implications

Taken together, mixed reality systems should minimize manual configuration during critical procedural phases and avoid interaction schemes that add fine motor burden or attentional switching. For first-in-human surgical guidance platforms, evaluation should assess not only whether commands are technically successful but also whether interaction remains reliable, safe, and low burden under sterile, time-pressured conditions without impairing procedural control [5,13]. Such interaction demands may also increase cognitive workload by requiring users to divide attention between holographic content, interface management, and ongoing procedural tasks. Future evaluation should therefore consider whether interaction remains manageable in cognitive as well as motor terms under realistic operative conditions. In our view, this is a prerequisite for meaningful clinical translation rather than a later-stage usability refinement.

Environmental Resilience as a Prerequisite for Clinical Adoption

Noise

Operating room noise is a clear challenge for voice-based interaction [13]. Across multiple studies, voice control has failed because of either poor command recognition or background noise [3,11]. This is especially important for surgical environments, where suction, drills, monitors, and team communication are routine features of the setting [3].

In our view, this issue illustrates why apparently attractive interface solutions may prove fragile in practice. Voice control may work well in controlled demonstrations, but its value in the operating room depends on resilience to realistic acoustic conditions. Future human factors studies should therefore test voice-based interaction under typical theater noise rather than under quiet laboratory conditions.

Lighting and Glare

Lighting conditions are repeatedly identified as important to usability and technical performance [5,6]. The 2025 AR-HMD review emphasized that displays may suffer from veiling glare and that lighting conditions should be incorporated explicitly into performance evaluation because they affect both image visibility and tracking reliability [6]. It further described how strong operating room lighting can compromise visibility and the accuracy of holographic matching tasks [6].

These are not merely optical inconveniences. They directly affect whether information can be seen, interpreted, and trusted at the point of care. In our view, future studies should regard lighting variability as a standard part of human factors evaluation, particularly for systems intended to present critical spatial guidance during surgery.

Occlusion and Line-of-Sight Disruption

Occlusion is one of the most consistently cited environmental and technical barriers [5]. In optical see-through systems, occlusion affects both visual presentation and tracking robustness, particularly when marker-based solutions rely on an uninterrupted line of sight [5]. The operating room’s crowded and dynamic nature means that staff movement, instruments, smoke, or blood may disrupt tracking or reduce confidence in overlay stability [5].

This is highly relevant for clinical translation because such disruptions are routine rather than exceptional. In our view, systems intended for real intraoperative use must be tested against these ordinary environmental disturbances. Otherwise, there is a risk of overestimating performance in idealized settings and underestimating practical failure modes in the theater.

Real-World Operating Room Complexity

Recent evaluation work has criticized idealized testing environments and stressed the need for ecological validity [4,6]. The 2025 review noted that many current experiments fail to replicate physiological movement, environmental lighting changes, marker interference, and operating room personnel movement [4]. Similarly, the optical see-through review argued that context adaptation remains insufficiently addressed in many systems [4,13].

In our view, this is one of the most important methodological gaps in the field. If mixed reality guidance is to be translated safely into surgery, then testing must move closer to the realities of clinical practice. Human factors evaluation should therefore include realistic environmental complexity as a design condition rather than treat it as a downstream implementation issue.

Implications

Environmental resilience is central to surgical usability [3,13]. A mixed reality system that performs well in static laboratory conditions may still fail clinically if lighting, noise, or crowding degrade display visibility, tracking, or input performance [6]. In practice, these environmental stressors rarely occur in isolation. Noise, glare, movement, crowding, occlusion, and changing lighting conditions may co-occur in the operating room, and their combined effects may shape display visibility, tracking reliability, communication, and user performance more substantially than any single factor considered alone. In our view, future evaluation should treat environmental resilience as a systems property rather than as a set of separate technical stressors.

Implications for Research and Clinical Translation

The central argument of this Viewpoint is that the safe and effective adoption of mixed reality in surgery depends on far more than registration accuracy or visual overlay performance alone. Clinical translation will be shaped by whether these systems can accommodate physical user realities, support reliable low-burden interaction, and remain functional within the environmental complexity of the operating room. These issues also have important implications for patient safety. For systems intended to support intraoperative decision-making, failures in perception, calibration, interaction, communication, or environmental robustness may affect how information is interpreted, whether uncertainty is recognized in time, and how safely clinicians act on device-derived guidance [5,6]. In practice, such failures may contribute to workflow disruption, procedural error, delayed fallback to conventional methods, or misplaced confidence in inaccurate or unstable guidance. Human factors evaluation should therefore be linked not only to usability and adoption but also to safe clinical decision-making and avoidance of patient harm [6,14].

These barriers are not confined to a single domain. Physical tolerability, dexterity and motor demands, and environmental resilience interact in ways that shape overall usability and trust in the system. A device may offer promising visual guidance yet still prove impractical if headset weight causes fatigue, if interaction methods interrupt procedural flow, or if performance degrades under theater lighting, movement, or noise. This shifts attention away from isolated technical optimization and toward the conditions required for reliable use in practice.

This perspective has several immediate implications for future human factors research. Studies of physical characteristics should move beyond generic usability reporting and include explicit consideration of comfort, visual demands, headset fit, fatigue, and compatibility with eyewear, masks, caps, and routine theater practices [3,11]. Studies of dexterity and motor skills should assess not simply whether interaction methods function, but whether they remain efficient, intuitive, and safe during real workflow and under sterile conditions [5,13]. Studies of environmental factors should be conducted in realistic or high-fidelity operative settings, including bright lighting, staff movement, background noise, crowding, and partial occlusion [6].

A systems human factors perspective also draws attention to several underexplored questions [8,9]. These include how system failure or uncertainty is recognized during use, what fallback processes should be triggered when mixed reality guidance becomes unreliable, who assumes responsibility when the system cannot be used as intended, and how shared understanding is maintained when only 1 operator has direct access to the display. This last issue is especially important in the inherently team-based operating room, where asymmetric access to holographic content may affect communication, shared situational awareness, supervision, coordination, and collective decision-making. Where device-derived guidance is visible only to the primary user, important cues may need to be verbally translated or otherwise mediated, increasing coordination burden and altering information flow.

These questions have practical implications for evaluation. Beyond individual usability, future studies should examine how mixed reality systems affect workflow disruption, team communication, shared situational awareness, workload, and failure recognition and recovery under realistic operative conditions. The choice of methods should depend on the procedural context and stage of development, but evaluation should move beyond generic usability reporting alone. Relevant approaches may include workload measures such as the NASA Task Load Index [15], usability instruments such as the System Usability Scale [16], situational awareness assessments, team communication measures, and structured assessment of error recognition and recovery. These practical priorities for evaluation are summarized in Table 1.

Governance and regulatory readiness also require greater attention [8,9,14]. For systems intended to support intraoperative judgment, accountability, clinical responsibility, supervision, escalation, and medicolegal responsibility are not peripheral implementation issues but core conditions of safe adoption. This is particularly important where device-derived guidance may be uncertain, unavailable, or in tension with operative judgment. In such cases, clear arrangements are needed regarding who retains decision-making responsibility, how uncertainty is recognized and communicated, and what governance structures support safe implementation and oversight during early clinical use [10,14].

In our view, human factors are still too often treated as secondary outcomes rather than used to shape system design and translational readiness from the outset [3,5,17]. Published work has also tended to focus more heavily on user-device interaction than on the wider team, organizational, training, and governance issues that are likely to determine real-world implementation. This concern is consistent with broader perspectives in medical AI, which emphasize that emerging AI-enabled technologies should be evaluated with the same rigor as other clinical interventions and implemented through transparent, patient-centered, multidisciplinary frameworks [14]. For mixed reality surgical guidance systems, particularly those intended for first-in-human use, this implies that user-centered design, context-of-use definition, and use-related risk assessment should be treated as core development requirements.

The limiting factor in clinical translation is no longer only whether mixed reality can function in principle but whether it can be integrated safely and practically into real surgical work. This implies that future studies should evaluate mixed reality not only as a display technology but also as a sociotechnical intervention shaped by surgical workflow, environmental complexity, and organizational conditions and guided by the human factors priorities set out in this Viewpoint. A systems perspective further suggests that future research should not be limited to refining user-device interaction alone. Failure recognition, fallback and takeover procedures, responsibility, team communication, shared situational awareness, training, competence, supervision, and governance are all part of the broader work system in which mixed reality guidance would be used. At present, these higher-level issues appear less well represented in the literature than ergonomic, perceptual, and interface-related concerns. Addressing this imbalance will be essential if mixed reality surgical guidance is to progress from proof-of-concept feasibility to safe and sustainable clinical implementation. Without this shift, mixed reality in surgery may remain technically impressive but operationally fragile, limiting progress toward safe, usable, and clinically sustainable adoption.

This Viewpoint has limitations. It does not address all human factors domains in equal depth, and some of the safety implications discussed are inferred from human factors principles rather than established through direct comparative clinical evidence. Even so, these issues remain important for technologies intended to support intraoperative decision-making and merit more explicit evaluation in future work.

Conclusions

Physical characteristics, dexterity and motor demands, and environmental conditions are central to whether mixed reality systems can be used effectively in surgery. Although mixed reality shows promise in improving spatial awareness, information access, and attention to the operative field, these potential benefits remain constrained by ergonomic limitations, perceptual challenges, interaction burden, and limited robustness in real operating room settings.

The next phase of development should therefore place human factors evaluation alongside technical validation as a core requirement for successful clinical translation. For mixed reality surgical guidance systems, particularly those intended for patient-specific intraoperative use, future studies should explicitly address physical accommodation of diverse users, low-burden and sterile interaction, reliable performance under realistic environmental conditions, and the broader system conditions required for safe communication, training, supervision, and implementation. Human factors research must therefore move beyond technical accuracy alone and help define how these systems can become safe, usable, and clinically adoptable in practice.