This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on http://humanfactors.jmir.org, as well as this copyright and license information must be included.
Eye-tracking technology has been used to measure human cognitive processes and has the potential to improve the usability of health information technology (HIT). However, it is still unclear how the eye-tracking method can be integrated with other traditional usability methodologies to achieve its full potential.
The objective of this study was to report on HIT evaluation studies that have used eye-tracker technology, and to envision the potential use of eye-tracking technology in future research.
We used four reference databases to initially identify 5248 related papers, which resulted in only 9 articles that met our inclusion criteria.
Eye-tracking technology was useful in finding usability problems in many ways, but is still in its infancy for HIT usability evaluation. Limited types of HITs have been evaluated by eye trackers, and there has been a lack of evaluation research in natural settings.
More research should be done in natural settings to discover the real contextual-based usability problems of clinical and mobile HITs using eye-tracking technology with more standardized methodologies and guidance.
Health information technology (HIT) systems are promising tools for improving quality, patient safety, and efficiency in health care systems [
Eye-tracking technology has been used to measure cognitive processes since the 1970s [
Two important measurements of eye-tracking technology are fixation and saccade [
Eye-tracking data can be integrated, synthesized, and visualized using software suites, such as commercially available analysis tools. Different types of visualizations, such as heat maps and gaze plots, communicate different types of information [
Based on the International Organization for Standardization (ISO) Standard, usability is the extent to which users can achieve a goal effectively, efficiently, and with satisfaction [
We conducted a systematic online database search to identify articles published before September 2014 that were relevant to the aims of this study. Articles were included as indexed in four reference databases: Medline, Web of Science, ScienceDirect, and PsycINFO. Broad keyword searches were used to identify relevant articles in each database. Each initial search focused on one of three key components: (1) a word or phrase related to usability evaluation, (2) a word or phrase related to HIT, or (3) a word or phrase related eye-tracker technology.
Keywords related to usability evaluation included usability testing, user experience, user test, user-centered design, system design, interface design, and interaction design. Keywords related to HIT included health IT, health informatics, health technology, medical technology, eHealth, telemedicine, communication tools, educational technology, decision support technology, health app, and wearable technology. Keywords related to eye-tracker technology included eye-tracking technology, eye tracker, Tobii, Sensomotoric Instruments, eye movements, gaze, eye fixation, and saccade. We also identified potentially eligible articles by manual literature searches, by examining article reference lists and by searching in Google Scholar.
We initially defined the scope of the review by determining inclusion and exclusion criteria. Papers were included if they contained all of the following: (1) the research used eye-tracking technology as a data collection tool, (2) the research evaluated an HIT with users, and (3) the research explicitly mentioned the improvement of HIT usability based on the eye-tracking data. Papers were excluded if they (1) were not in English, (2) were published 10 or more years ago (ie, prior to 2004), (3) did not evaluate an HIT, (4) focused on technologies not related to health care, (5) used eye-tracker technology for some purpose other than data collection (ie, as an input device), or (6) did not mention any indications of the system usability based on the eye-tracker data.
Based on the methods-description approach, we analyzed the selected papers that met the inclusion criteria [
A total of 5248 papers were found by using the search terms and databases described above. Of these, 1888 papers were removed due to duplication. After reviewing the titles and abstracts based on inclusion and exclusion criteria, we eliminated another 3351 papers. This resulted in a total of 9 papers remaining for this review (see
All selected papers discussed user evaluation of a type of HIT using eye-tracking technology as a data collection tool. Of the 9 papers, 3 of them (33%) mainly discussed a usability evaluation of an HIT using eye-tracker technology [
Summaries of papers used in the review.
Author and reference | Title | Purpose | Key findings |
Bansback et al [ |
Development and preliminary user testing of the DCIDA (Dynamic computer interactive decision application) for ‘nudging’ patients towards high quality decisions | To develop and test a computer application that enhances conventional patient decision aids so that common decision errors made by patients can be reduced. | The Dynamic Computer Interactive Decision Application (DCIDA) version of patient decision aids was understandable to users and it was able to help users focus on attributes that are of individual importance to them. |
Barkana and Acik [ |
Improvement of design of a surgical interface using an eye tracking device. | To use eye-tracking technology to improve the design of a surgical interface to obtain the optimum configuration. | The interface of the early version of a surgical interface was redundant. With two larger scans at higher spatial resolution on the interface, participants were able to complete tasks more quickly, and the visual acquisition corresponded more to the natural visual search. |
Eghdam et al [ |
Combining usability testing with eye-tracking technology: evaluation of a visualization support for antibiotic use in intensive care | To investigate if Infobiotika supports efficient and effective navigation and to observe the user's navigation paths, visual scan patterns, and distribution of visual attention. | Infobiotika was effective and efficient in terms of navigation support, and was a learnable product for intensive care unit (ICU) physicians. |
Forsman et al [ |
Integrated information visualization to support decision making for use of antibiotics in intensive care: design and usability evaluation | To investigate the role of visualization as a method to support intensive care physicians’ decision making about antibiotic use, analyze users’ work processes and information needs, develop an interactive tool for integrated information visualization, and perform usability testing. | The visualization tool was usable for supporting ICU physicians in antibiotic use. Physicians had increased awareness of a patient's infection-related data and felt more in control of the situation. |
Henneman et al [ |
Providers do not verify patient identity during computer order entry | To determine the frequency of verifying patient identity in an emergency department (ED) during computerized provider order entry (CPOE). | Medical providers did not usually verify patient identity prior to selecting the patient from the list and ordering tests. They often did not recognize patient-identity errors in the system. |
Kules and Xie [ |
Older adults searching for health information in MedlinePlus – an exploratory study of faceted online search interfaces | To examine how searchers interact with a faceted Web-search interface. | Faceted interfaces played a substantial role in participants' use of the search result pages. The severity of the health condition affected the use of faceted interfaces. |
Liu et al [ |
The use of illustration to improve older adults’ comprehension of health-related information: Is it helpful? | To examine whether explanatory illustrations can improve older adults' comprehension of written health information. | Older adults had difficulties understanding the illustrations as well as integrating the illustrations with the text. Older adults did not benefit from the use of illustration. |
Rashid et al [ |
Preliminary usability testing with eye tracking and FCAT analysis on occupational safety and health websites | To measure effectiveness, efficiency, and satisfaction of the Occupational Safety and Health (OSH) website, and to gather user feedback. | Eye-tracker data and user feedback helped identify usability problems of three OSH websites. |
Wolpin et al [ |
Development and usability testing of a web-based cancer symptom and quality-of-life support intervention | To develop a user-centered prototype, and assess user preferences from usability testing of a revised prototype of the Electronic Self-Report Assessment for Cancer-II (ESRAC 2.0) project. | An application was developed that integrated the patients’ needs through the methods of participatory design, usability testing, and iterative development. |
Flow diagram of the study selection process.
We identified different research questions that are answered by eye-tracker technology in the selected papers. The first question that can be answered by eye-tracker technology is whether the user experience and performance using an HIT has been improved based on the eye-gaze patterns, which primarily reflects the system effectiveness, efficiency, and user satisfaction [
We identified different types of HITs in the selected papers. In terms of functionality, the technologies included online health information website interfaces [
We identified different experimental apparatuses of the user tests. Researchers evaluated HITs in the forms of developed computer website/application [
Out of the 9 papers, 2 of them (22%) reported a failure to collect eye-tracking data during the usability test [
We identified three basic eye-tracker measurements in our selected papers. The measurements included fixation duration [
The selected papers also showed other usability evaluation methods that are combined with eye-tracker data to explore usability problems in HIT systems. The methods include the System Usability Scale (SUS) [
Summary of research questions.
Questions and answers | Reference | ||
|
|
||
|
|
System effectiveness, efficiency, and user satisfaction | [ |
|
|
Linkage between human visual stimulus and cognitive processing | [ |
|
|
The variability of human performance | [ |
|
|
||
|
|
|
|
|
|
Health information website interfaces | [ |
|
|
Surgical interfaces | [ |
|
|
Decision support systems | [ |
|
|
Computerized provider order entry systems | [ |
|
|
Symptom and quality-of-life information systems | [ |
|
|
|
|
|
|
General public | [ |
|
|
Patients | [ |
|
|
Physicians | [ |
|
|
||
|
|
||
|
|
Developed computer program | [ |
|
|
Simulated prototype | [ |
|
|
Screenshots | [ |
|
|
|
|
|
|
On-screen eye tracker | [ |
|
|
Mobile eye tracker | [ |
|
|
Head-mounted eye tracker | [ |
|
|
||
|
|
|
|
|
|
Fixation duration | [ |
|
|
Eye movement location | [ |
|
|
Fixation count in area of interest | [ |
|
|
|
|
|
|
Heat map | [ |
|
|
Gaze plot | [ |
|
|
Statistical analysis | [ |
|
|
||
|
|
Think-aloud protocol | [ |
|
|
System Usability Scale | [ |
|
|
Questionnaire | [ |
|
|
Posttest interview | [ |
|
|
Metrics measurement | [ |
|
|
Feedback Capture After Task | [ |
The purpose of this literature review was to examine usability evaluations of any type of HIT using eye-tracking technology. This review also aimed to identify the research gap and potential uses of eye-tracker technology in future HIT research. This review was conducted based on the inclusion and exclusion criteria specified in the Methods section. Based on the results, we determined that, although eye trackers provide rich data for the improvement of HIT systems, the use of eye trackers for usability evaluation of HITs is still in its infancy, as only 9 papers were found that fit within the inclusion criteria.
We organized the results into five main questions: (1) What research questions are answered by eye-tracking technology?, (2) What types of health care information technologies were evaluated using an eye tracker?, (3) What was the experimental apparatus of usability evaluation?, (4) What did the eye trackers measure and how was data analyzed?, and (5) How was eye-tracker technology combined with other usability methods?
Papers that were included in this review had different purposes and research goals. The types of HITs evaluated were limited, resonating with our finding that the use of eye trackers for the evaluation of health IT is in an early stage. However, eye trackers are becoming a promising tool for usability studies, as demonstrated by the increasing number of research studies in recent years. We also found that researchers used various means of data collection and analysis using eye trackers. On the one hand, this demonstrates the rich variety of data that can be captured by eye trackers and the flexibility of interpretation of eye-tracker data. On the other hand, it shows the lack of a consensus on how to conduct user evaluation of HITs using eye trackers at this stage. In addition, we found that eye-tracking technology, as a part of usability evaluation methodology, was supplemented by other traditional methods. Generally, eye-tracking data can reveal the patterns of user difficulties when completing tasks using HIT, while other supplemental inquiries are used to unfold the reasons behind those patterns. Therefore, eye-tracking technology has to integrate with other techniques, as most physiology measurements do, because eye-tracking technology alone cannot tell the entire story.
The reviewed papers reflected different research questions that were answered by eye-tracking technology. Of the 9 papers, 6 of them (67%) were directly related to the system usability, focusing on the efficiency, effectiveness, and satisfaction when completing tasks with a specific HIT. Of these, 1 paper was related to the examination of a gap between visual and cognitive process. For example, a user missed information because he/she did not pay attention to it, even if eye-tracking technology suggested that the user had seen that information [
The reviewed papers involved five different types of HITs, including 3 out of 9 papers (33%) evaluating health information website interfaces, 3 papers (33%) evaluating decision support systems, 1 paper (11%) evaluating a surgical interface for physicians, 1 paper (11%) evaluating a computerized provider order entry system for physicians, and 1 paper (11%) evaluating a symptom and quality-of-life information system for physicians. The reviewed papers involved three different types of users, including 3 papers (33%) for general public health IT, 2 papers (22%) for patients, and 4 papers (44%) for physicians. Thus far, eye trackers have been used most often to evaluate health information website interfaces. This indicates that evaluating a website interface using eye-tracker analysis may provide rich theoretical guidance and reveal available practices that researchers can refer to [
However, there is much potential for eye-tracker technology to be applied to other types of health IT as well. One particular aspect of health IT that lacks usability research using eye-tracker technology is electronic health record (EHR) systems. EHR systems have helped to revitalize physician and nursing practice, and have the potential for positive impact on clinical processes in terms of efficiency, productivity, and patient safety [
Another gap exists in the application of eye-tracking technology to usability studies of novel consumer HITs. Health apps and devices are becoming prevalent in the market. Devices such as the iPad, iPhone, iPod Touch, and Apple Watch have been the target devices to provide a richer and more convenient user experience of health care information technology [
In terms of the prototypes that were evaluated in the reviewed papers, a majority of them (4/9, 44%) evaluated developed computer programs that have already been adopted in health care systems. Of these papers, 2 of them evaluated screenshots of the real websites, and 2 of them evaluated usability using a simulated prototype in high fidelity. Unlike many other usability techniques, such as the formative usability evaluation approach, that are primarily introduced in the early phase of the user-centered design process, we found almost all the papers evaluated HITs at a very late phase or even after implementation, as a summative approach. The benefit of doing a summative usability evaluation is that researchers are able to create an approximation of the actual use scenario of HITs. Compared with low-fidelity, nonfunctional prototypes, such late-phase testing is more likely to uncover real usability problems [
Unfortunately, at this point there has been no usability evaluation conducted using eye-tracking technology in real settings. All of the reviewed papers conducted user studies in a meeting room or a usability lab. Possible reasons for this gap could be the mobility limitations of eye-tracker technology, the possible intrusion of such technology on work, technical difficulties, and the calibration process of the eye-tracking equipment. Nearly half of the papers (4/9, 44%) used eye trackers that were embedded within a computer screen, which are impossible to move into real settings. Of the 9 papers, 2 of them (22%) used head-mounted eye trackers, which are easy to move but intrusive to the health care provider’s work if the evaluation is conducted in the field, negatively influencing the work in that time-sensitive environment. Moreover, such head-mounted trackers are more likely to have technological difficulties, which risk accurate data collection. Of the 9 papers, 3 of them (33%) used mobile eye trackers, which are probably the best equipment to be incorporated into field research in the real HIT setting. However, the calibration process may add additional steps to the already complex workload of nurses or physicians. Moreover, it is unlikely that a health care provider will stay in one place for a long period of time, and their movements will disrupt the calibration [
We found that the eye-tracker measurements in the reviewed papers were mainly fixation and saccade, which supports the finding by Poole and Ball [
Qualitative analysis is becoming prevalent partly because of the improvement of software suites, making the analysis easier and less intensive. Of the 9 papers, 3 of them (33%) translated the data into qualitative visualization, such as heat maps and gaze plots. While statistical analysis is powerful in comparing completion time and errors, it is only part of the usability evaluation. For a full usability evaluation, we believe the qualitative data in visualization can illustrate more usability problems. Using a heat map, it is easy to determine if specific content is usable or not. Using a gaze plot, it is possible to determine if users follow an efficient and predetermined route when searching for specific information on the interface.
However, we found that the interpretation of such visualizations lacked scientific guidance based on an established theoretical method, so interpretations tended to seem arbitrary and subjective. At this point, researchers are struggling to find a theory or a commonly used procedure to guide the interpretation of heat maps and gaze plots. Therefore, we expect that in the future a more structured system of interpretation will be developed for heat maps and gaze plots.
With the visualization of eye-tracker data, researchers can identify the areas of an interface that have created difficulties in participants’ minds. However, based solely on the eye-tracker data, there is no way to understand the precise cognitive reasons behind a participant’s eye-gaze patterns. For example, there might be many possibilities for an eye fixation, such as fatigue, distraction, confusion, and engagement [
It is interesting that there are three different think-aloud methods used in the reviewed papers: concurrent think aloud, RTA, and a hybrid of both. The concurrent think-aloud method is the traditional method widely accepted and applied by usability evaluation researchers. It is a method that asks participants to verbalize their thoughts while interacting with the system [
Another reviewed paper used a hybrid of concurrent think-aloud and RTA methods [
Although eye tracking is a promising technology, the application of eye-tracking technology to health IT usability evaluation is still in its infancy, with limited theoretical guidance and practice. Therefore, we reviewed papers that were related to usability evaluations of HIT using an eye tracker, to understand the current state, identify the gaps, and envision future research. There is no doubt that eye-tracker technology would be able to provide valuable data if well-integrated with other traditional usability evaluation methodologies. However, the lack of field research of clinical and mobile HITs in natural settings is a huge gap that needs to be filled. Scientific guidance is also needed for the interpretation of eye-tracking visualizations. Eye trackers can play a significant role in the future of usability evaluations of HIT if they are used effectively and correctly.
computerized provider order entry
Dynamic Computer Interactive Decision Application
emergency department
electronic health record
Electronic Self-Report Assessment for Cancer-II
Feedback Capture After Task
health information technology
intensive care unit
International Organization for Standardization
National Aeronautics and Space Administration Task Load Index
Occupational Safety and Health
retrospective think aloud
Sensomotoric Instruments
Short Post-Assessment Situational Awareness
System Usability Scale
This research was supported by the Center for Patient Care and Outcomes Research at the Medical College of Wisconsin. We would also like to thank Beth Schewe for her contribution to the paper.
None declared.