The health care system in the United States is currently undergoing extensive changes. Possible causes for such challenges faced by its delivery system are increased demand for health care due to an increased number and changing lifestyle leading to an increase in chronic diseases, the demand for increased accessibility of care outside hospitals, moving health services into the patient’s own homes, the need for increased efficiency, individualization and equity of quality-oriented health care with limited financial resources, the difficulties of recruiting and retaining personnel in health care services in general, and in home and elderly care in particular [1,2]. Telehealth, the use of electronic information and telecommunication technologies to support long-distance clinical health care, patient and professional health-related education, public health and health administration [1,3], has the potential to address these issues. One subsection, telemedicine, the use of technology to provide and support health care when distance separates the clinical service and the patient, appears to be particularly attractive [4].

Although playing an important role in addressing the health issues of patients living in underserved and rural areas, telemedicine is now attracting attention beyond these limited regions. It offers mechanisms for centralizing specialists, reducing costs for specialty care, and supporting primary care clinician needs in the urban and suburban areas these typically serve [5-9]. The possible benefits of using these systems have resulted in an increased interest in telemedicine. For example, they help patients with chronic illness and those with limited mobility to connect with a health care facility from the comfort of their homes [1], and it is important because it reduces the stress for patients who otherwise would have to travel long distances for their appointments [10]. Currently, this remote care is extensively used for clinical visits that do not require physical presence such as behavioral health [11,12], counseling [13-15], follow-up [8,16], and patient education [17], with studies finding telemedicine an appealing solution for the real-time remote monitoring of patients. It has also shown promise for improving patient knowledge of health care, thus helping them better manage their diseases or illnesses. With the expanding technical capabilities and the decreasing costs of telemedicine software solutions, home-based telemedicine is becoming more widely used, evidenced by a recent workshop conducted by the National Academies that discussed the potential of scaling such delivery of care for a growing number of patients [2,18].

Unlike face-to-face encounters, in which clinicians and patients are both located in the same setting, telemedicine participants usually use the teleconferencing systems at their respective locations. Thus, for a telemedicine system to become widely accepted, it should not only be functional but also user-friendly [19-23]. The Institute of Medicine (IOM) recently emphasized the role of usability in telemedicine systems, given its potential to replace regular clinical visits which are both time-consuming and resource-demanding [2,8,23].

However, limited research has been conducted evaluating the perceived usefulness and usability of such tools from a home-based video telemedicine system perspective [24,25]. The evaluation of a user interface can be carried out by 4 ways; formally using analytical tools, automatically using computer technology, empirically, that is, testing with users, and heuristically [25-27]. Heuristic evaluation is the process of usability testing wherein evaluators are provided with an interface and asked to comment on it based on a set of heuristics [27,28]. The efficiency of this system of evaluation allows for an iterative design process of user interfaces [18]. Studies have found that this type of evaluation can reveal both major and minor usability issues, including problems that lead to errors and user dissatisfaction content [29-31]. Furthermore, it has been extensively used to ascertain usability issues in the medical field ranging from websites to medical devices [25,32-34] to health information technology applications [35-40]. In light of these advantages of a heuristic evaluation, this study aimed to understand the issues of the clinician’s interfaces of 4 telemedicine platforms. The issues uncovered through this heuristic evaluation could serve as a basis to improve the clinician’s interface in telemedicine platforms.

The criteria for a telemedicine system to be included in this heuristic evaluation were as follows: (1) the system is primarily used to deliver video-based telemedicine at home; (2) the system does not require specialized or proprietary equipment for home use; (3) the system runs on an Internet-connected computer with audio and video capabilities; and (4) the system is Health Insurance Portability and Accountability Act (HIPAA) compliant, which aims at protecting the health rights and privacy of patiens [41]. Initially, the telemedicine systems used by the medical staff at the Medical University of South Carolina (MUSC) Center for Telehealth and South Carolina (SC) Telehealth Alliance were reviewed. Subsequently, 8 software applications, Adobe Connect, Cisco WebEx, Cisco Jabber, Doxy.me, Polycom, Skype, Vidyo, and VSee, were identified as potential candidates based on this preliminary review. Next, a detailed analysis of the features of each and its primary use were conducted. A total of 5 key stakeholders including the physicians and directors associated with MUSC Center for Telehealth and SC Telehealth Alliance were consulted. It was understood that Adobe Connect, Skype, Cisco WebEx, and Cisco Jabber could be used to deliver video-based telemedicine, but they were not currently used extensively for doing so. On the basis of this feedback, the telemedicine tools selected for this research were (1) Doxy.me, (2) Vidyo, (3) VSee, and (4) Polycom.

The investigation reported here was based on the experts’ heuristic evaluations and the severity of the problems based on a severity rating scale. A heuristic evaluation is a discounted usability method in which the evaluation of an interface is done based on established usability principles. Five human factors engineers—three PhD students, one assistant professor, and one master’s degree student, all with prior training in conducting heuristic evaluations—were recruited to serve as the subject matter experts for this study. All received verbal information about the purpose and the goal of the study, and a detailed written task flow to guide the evaluation of the 4 telemedicine platforms using a modified heuristic evaluation procedure. They were compensated with a US $20 Amazon gift card for their time.

The method of evaluation used in the study was a heuristic evaluation, a discounted usability evaluation method, combined with a severity rating scale [27,42]. Specifically, Nielsen’s heuristics were used because of their widespread use and acceptability [27,43-45]. The heuristics, which are listed in Table 1, were used to highlight possible usability issues.

A heuristic evaluation, typically conducted with 5 experts, detects up to 80% of the problems [18]. For this study, the experts individually conducted the assessment in a closed lab setting to avoid bystander bias. A 5-point severity scale was applied to each of the usability issues to indicate the level of concern [27]. The scale ranged from issues which may not impact the usability of the system to issues that could potentially lead to its failure. The 5-point scale is as follows [46]:

0—May not be a problem: other observers do not agree that this is a usability problem

1—Cosmetic problem only: it need not be fixed unless extra time is available

2—Minor usability problem: fixing it should be given low priority

3—Major usability problem: it is important to fix it, should be given high priority

4—Usability catastrophe: imperative that it is fixed before product can be released

Understanding the source of errors in a task begins with an in-depth understanding of the task flow [33]. This study, thus, began with a detailed task analysis for each of the 4 telemedicine systems to help understand the feedback they provide and the potential problems the user could face. This task analysis also included determining the knowledge the user must have in order to perform the task successfully. Before actual evaluation, the researcher discussed the detailed task analysis, heuristics, and severity ranking scale with the experts. As the experts were from the field of human factors and familiar with heuristic evaluation studies, only context-specific instructions were provided to evaluate the telemedicine platforms. A separate sheet containing the list of the heuristics and the severity rating chart was also given to the experts for their reference. The experts then evaluated the systems from the clinician’s perspective with the help of a hypothetical patient with whom no communication was carried out. After which, they listed the heuristic violations individually. The tasks to be completed by the evaluators were as follows:

Initiation: Create an account, log into the portal or desktop application, send an email invitation for the telemedicine session, call the patient.

Consultation: Toggle microphone and video, enter full screen mode, enter data into a chat box (where applicable), and end video call.

On completing individual evaluations, experts discussed their findings with others in a postevaluation debriefing session. In the case of extreme inconsistencies, the evaluators discussed and came to a consensus about the appropriate rating. Individual lists were subsequently compiled for data analysis. Figure 5 outlines the experimental procedure followed in this study which lasted approximately 1 h.

For the evaluation, each expert recorded the heuristic violations for the respective tasks, including grading the severity of the issues. The individual ratings from the evaluators were averaged to obtain a single value of the severity. These data were then compiled to understand which heuristics were most violated and the severity was analyzed to prioritize the problems. The number of heuristics violated was graphed for the telemedicine session initiation and consultation.

The heuristic evaluations of the experts for the clinician’s interface revealed a total of 46 unique issues: 11 for Doxy.me, 10 for Vidyo, 12 for VSee, and 13 for Polycom. Of these, 22% (10/46) concerns were recognized by all the experts. Tables 2-4 list the important usability issues and the heuristics they violated for initiating, conducting, and concluding a telemedicine session, respectively. Figures 6-8 are graphical representations of heuristic violations for initiating, conducting, and concluding a telemedicine session, respectively. As these figures show, 60.9% (28 issues) of the issues was identified in the initiation phase, 33% (15/46 issues) in the telemedicine session phase, and 7% (3/46 issues) in the conclusion phase. Sharing or setting up microphone and camera was one of the specific issues observed in the initiation phase (Figure 9). Using default email client (Microsoft Outlook) to invite patients was one of the issues identified during the telemedicine session. During conclusion, difficulty to find the log-out button was pointed out as an important issue (Figure 10).

The heuristics most frequently seen violated were visibility of system status and error prevention, each with 11 violations (24%, 11/46), with esthetic and minimalist design being second with 6 out of 46 violations (13%, 6/46). Violations were not observed for the heuristics user control and freedom and flexibility and efficiency of use. It was found that (1) 4 out of 46 (9%, 4/46) violations were recorded for each of consistency and standards, recognition rather than recall, help users recognize, diagnose, and recover from errors and help and documentation heuristics, and (2) 2 out of 46 (4%, 2/46) violations were observed for match between system and real world heuristic. Specific issues related to visibility of system status included lack of feedback while downloading setup (.exe), lack of saliency of notifications on receiving a message or when a patient enters a Web-based waiting room, and the absence of salient call end and log-out icons. Inconspicuous check boxes, inadequate labeling of icons, and failure to exit full screen on completion of a session were identified under error prevention.

The experts rated two issues as requiring immediate action. The multiple check-in options available in the welcome screen of Polycom was one, with the experts finding that the availability of multiple options confused the user, and that the welcome screen did not contain sufficient information to choose the appropriate device (Figure 11). The second problem highlighted by the experts was the use of a default email client to email invitations to the patients. Three (Doxy.me, Vidyo, and Polycom) of the four conditions redirect the user to the default email client to send email invitations to the patients. It may be more effective to give the user the choice of using the email client of his or her choice.

During the debriefing, experts discussed their most and least favorite aspects of each of the platforms. Experts indicated that they preferred interfaces that were not cluttered with too many options, language that they could relate to that in the real world, and systems that provided adequate and timely feedback for their actions. The least enjoyable aspects were welcome emails from the telemedicine platforms with multiple links, the use of a default email client to invite patients, and the failure of many options to respond the way expected.

The heuristic evaluation was conducted using a structured table containing the task flow and a column for experts to input the problems they found for each task, the respective heuristic violated, the severity of the violation, and possible solutions. Several issues were identified during the course of this evaluation. Visibility, error prevention, and minimalistic design issues were frequently violated. The effect of such issues on the ability of a user to process information can be explained using the information processing model [47]. A detailed description of the different aspects of the information processing model that are affected due to these issues would result in developing better solutions.

The information processing model [47] can be used to understand the impact of these issues on the user’s ability to understand and make decisions (Figure 12). This model illustrates the procedure of the human cognition process. The sensory register includes our sense organs that help a person take cues from our surroundings, which then leads to understanding or perceiving these cues. The working memory refers to the understanding and retention of information only for the span of completing a task. However, the long-term memory involves the retention of information for longer period of time such as a few weeks, months, or years. Using the information in the working memory and the long-term memory, the process of thinking and decision-making occurs to make a decision about the cues obtained from the environment. Once thoughts about the cues have been formed, an appropriate response to the cue is developed and based on this, an action is executed in response to the cue obtained. The execution of the response is again taken in by the sensory register and stored in the long-term memory for future situations. Throughout this process, there is also a constant requirement of attentional resources which help the user to focus on the necessary information and eliminate the rest.

On the basis of the issues specified, the lack of feedback would have a direct effect on the perception of the process. As a result, the user would have difficulty in deciding the subsequent process to be carried out in the procedure. It was also indicated that the popup for the chat box was not salient. This would directly affect the sensory register as the popup would not be visible and hence the user would fail to understand (perception) that a message has been received. The icon size and design would again affect sensory register and perception. The content of the email invitation, which was reported to require immediate attention, will prove to be an important issue affecting the working memory. The working memory, responsible for understanding and retaining information until the completion of a task, would be affected due to the large amount of information or the lack of information in the email invitation. Another problem reported as requiring immediate attention was the use of default email client to send email invitations to patients. This could potentially require the retrieval of passwords to log into the system which affects the long-term memory.

Although of lesser importance, there exist some other issues which must be studied with respect to their impact on a user’s decision-making. One such issue is the need to enter large amount of data for registration. This could affect the working memory limits of a person as they would be required to read and retain multiple data to enter. In three of the four platforms analyzed—Doxy.me, Polycom, and Vidyo—it was seen that the clinician was required to send an email invitation for every meeting. This would add to the working memory limits to process immediately available information and the long-term memory to remember patient name and email address to send the emails. The popups used to share the microphone and camera was indicated to be inconspicuous resulting in additional load on the sensory register due to lack of visibility.

On the basis of the understanding of the different areas of the information processing model affected by the issues and the issues highlighted by the experts, certain design recommendations were developed for telemedicine systems. Some of the key findings for improving the interaction of physician with the interface to enhance the usability of telemedicine platforms are given in Textbox 1.

or Apart from providing recommendations for improvement based on the information processing model, this study also demonstrates the practicality and ease of applying heuristic evaluation in usability studies. The entire process of conducting the study and analyzing the results took a week’s time and did not involve the use of any software applications. The efficiency of this method makes it well-suited for use during the early development stages [18].

This study is not without its limitations. In this study, we evaluated the telemedicine systems on a Windows 7 computer with Mozilla Firefox browser only. These systems need to be tested on multiple operating systems and Web browsers. Also, the evaluators for this study were not medically trained professionals. Future studies have to be conducted with actual clinicians to find usability issues from their perspective. Furthermore, as indicated by Nielsen and Molich [27], this heuristic evaluation like all others, only helps to identify the usability issues without providing solutions to address them.

Multiple studies have been carried out explaining the effectiveness of telemedicine in providing medical care with little research focusing on the ease and usability of these systems [24]. In this study, we used a heuristic evaluation and severity rating method to assess the usability of 4 telemedicine software platforms with a focus on understanding the interface issues faced by the clinician. Furthermore, the information processing model was used as the baseline to explain the impact of these issues on the user’s capability in making decisions. The heuristic evaluation and severity rating method was found to be effective in uncovering issues in the interface as 46 unique issues were uncovered across 4 different platforms. Prominent issues among these, whose impact was explained using the information processing model, is an indication of the need for further human factors concept-based studies of the interfaces of telemedicine systems.

With a focus on the clinician’s interface design, this heuristic evaluation was found to be an effective method for uncovering violations. This heuristic evaluation identified only potential usability problems in an existing interface; usability studies involving physicians could further indicate aspects of the system that work well and identify the most appropriate functionalities. However, with limited resources available, heuristic evaluation is a practical, affordable, and efficient method for revealing usability problems. Experts liked systems that had a straightforward and simple interface and that did not require installation. In addition, they preferred systems that sent simple welcome emails. From a telemedicine point of view, this is important as clinicians and technicians do not have the time to spend navigating and comprehending complex platforms.

Heuristic evaluation is a discounted usability evaluation method with limited generalizability. Future studies need to focus on detailed usability evaluation with actual clinicians and patients. Conducting retrospective interviews with the users help the designers understand their needs and in turn design or modify the system appropriately.