Nielsen [39] argues that, in exploring usability issues among homogenous users, the first 3 users will help discover problems in an exponential manner. The data are hypothetically saturated after the fifth user. In addition, the primary users of the current version of MEDIC are orthopedic specialists and physicians who normally provide consultations with each other. As a single-site study, the study site was an orthopedic clinic in a large medical school in Bangkok, Thailand, housing around 60 orthopedic specialists and physicians. The study enrolment was announced in the department meeting and all participants joined voluntarily. One orthopedist who regularly used the MEDIC app was recruited to be the control participant. Four specialists and 1 resident who had none or a few experiences of using MEDIC were recruited in this study. To control the complexity of the task and the variability of the platform, the usability tests were conducted using the MEDIC iPhone operating system platform only. Therefore, all participants had to be current iPhone users. The study protocol (IRB 756/62) was approved by the Institutional Review Board, Faculty of Medicine, Chulalongkorn University.

This study was conducted in a controlled setting in the Clinical Skill and Simulation Center (Figure 1). Upon arrival, participants were introduced to the MEDIC app through a 5-minute video presentation providing information on the main features and functions of the app. Thereafter, participants were asked to complete a usability test session. The overall test for each participant took about 90 minutes to complete. There were 5 tasks given to the participants to complete individually. In the context of an orthopedic clinic, these tasks were designed to cover the basic functions of the app and the data capturing process in simulated clinical situations. The participants were required to use the app on the provided iPhone 8. They were also allowed the use of other apps on the device to complete the tasks. Owing to the collaborative nature of the teleconsultation work, the users had to create a private group serving as a sharing space to communicate between physicians. The group could be utilized for a clinical unit, a discussion about a specific case or a group of cases, a research project, or a certain task force. Thus, the first task assigned the study participants to create a group and invite other users to join the newly created group. Since MEDIC is designed to be a patient-focused teleconsultation app, MEDIC allows physicians to create and manage patient records within a group space. Only physicians in the group can view and manage patient records in the group. Therefore, the participants were asked to create a new patient record in task 2. In addition, to evaluate how physicians use the app to manage patient records in an environment similar to the natural setting, the study participants were asked to collect patient information by using the app in a simulated clinical visit in task 3.

Another key function of MEDIC is to allow physicians to communicate in a standardized manner via the data collection form. There are several scales and measures that are essential for the clinical management of patients. Some are standardized, while some are tailored and customized within a group. The data collection form function allows physicians to create and customize a form to be used among the physicians in the group. Therefore, task 4 was designed to investigate how the participants used the data collection form by asking them to create a form to collect a Mangled Extremity Severity Score, which is one of the most common scales used in orthopedic clinical practices. Although the first 4 tasks aimed to observe how the participants created and managed data within the app from a sender perspective (eg, medical students, resident physicians, referrers), the last task was designed to evaluate the usability of the app from the perspective of the receiver (eg, peers, advisors, supervisors, referees). The participants were asked to review existing patients’ records who were asked for consultation. The participants were asked to provide clinical opinions as well as to enter a diagnosis and treatment plan in the consulting case’s record. While performing the tasks, participants were asked to speak aloud their thoughts to the researchers. Participant activities were logged using the video camera and screen recording function of the iPhone. The description of all the tasks is shown in Textbox 1.

Although there was no limitation on the completion time, participants could leave tasks incomplete at any point in the task. The radiograph used in task 5 is shown in Figure 2. After completing each task, the participants were interviewed using a semistructured interview approach to obtain detailed information regarding their behavior and experience. Additionally, an exit interview was also conducted after all the tasks had been completed. All user actions were recorded using a screen capturing app and video recording. One of the research team members also observed the participants and recorded their actions in an observational form.

The data from the think-aloud protocol and the semistructured interviews were transcribed. The observation notes were validated with the video recording. Quantitative data, for instance, time, number of clicks, and number of screens used, were recorded in MS Excel and analyzed using descriptive statistics by comparing with expert users’ performance. Thematic analysis using a deductive coding approach was adopted to analyze the transcripts, observation notes, and screen and video recordings by using the qualitative data analysis software, NVivo version 12 (QSR International). The primary coding scheme adopted PACMAD usability attributes [18], including effectiveness, efficiency, satisfaction, learnability, memorability, error, and cognitive load. The coding was conducted by 2 assessors independently. The codes and categories were then compared. Any disagreements were resolved by discussion between the 2 assessors.

To explore the usability issues of MEDIC, Table 1 shows the measures collected during and after tasks. To triangulate the results, the data were derived from 3 data sources: observation, think-aloud responses, and interviews.

It is apparent that certain measures were attributed to more than one usability domain, for instance, perceived task difficulty addresses both learnability and cognitive load. The time used to accomplish assigned tasks was also used to evaluate learnability and efficiency. For cognitive load, although the NASA Task Load Index is normally recommended [18,40], we considered Flood’s hypothetical approach [41] instead since it specifically addresses the cognitive load in a mobile environment in the context of clinical and health care practices. Furthermore, 2 additional measures were collected to understand user characteristics: (1) familiarity with each task assigned (rating on 1-7 Likert scale) and (2) familiarity with heavy-loaded tasks (eg, writing, filling out a form) on mobile platforms.

MEDIC, developed by Deverhood, Thailand, is a smartphone teleconsultation app for physicians to communicate with each other in various settings. As the app is designed to support patient-centered care, the main features of the app include patient medical data such as medical history, physical examination, clinical images, and diagnostic questionnaires. Health care providers can access, collect, and modify data from both desktop and mobile platforms, including both iPhone operating system and Android. However, the testing version in this study was on the iPhone operating system platform to control the environment. All data were to be uploaded to the cloud server; therefore, an internet connection was required while using the app. Designed to support collaboration among physicians and specialists, the MEDIC interface is divided to support 4 main tasks, namely, forming a team, data form creation, data recording, and data reviewing. The first task begins with forming a team such as a research group or a multicenter collaboration by creating a group and adding members. To invite members to the group, all teammates must have accounts with MEDIC. Group members can be removed or included by the group administrator.

Although MEDIC has been designed to collect generic patient records (eg, demographic, diagnosis, medical history, treatment), the app also allows physicians to create a data form to support their specialty, such as a case record form and functional score. However, licensed questionnaires should be authorized by the licensed owner in advance. There are 10 data input formats that can be used in the form: check boxes, drop-down lists, multiple choice, linear scale, multiple choice grids, free text, number text, date-time, picture, and video link. The content of the form can be organized into a section. Each question can be set as a required status, which should be completed or the form cannot be submitted. Relevant forms should be assigned to the related group. The data capturing process begins with patient registration with the group by entering a general profile and collecting data using a free text box, form, and camera tool. All recorded data can be reviewed by pressing the previous history tab, which shows free text history, forms, and images that were recorded in the past. Apart from all the main features, this app allows users to fill out their profiles for reference and a setup passcode lock to increase data security. The main features of the app are shown in Figure 3.

Apart from the capability of clinical data capturing, MEDIC is suitable for teleconsultation between health care personnel. The MEDIC app provides organized information, including history, laboratory findings, and clinical and radiological images, and patient condition and management are automatically sorted in a chronological order so that it is convenient for reviewing disease progression and treatment plan. Patient data privacy protection is improved by using MEDIC instead of social networks such as WhatsApp or Facebook messenger because the data access is limited to authorized persons for use in patient management.

The following section reports different measures covering all 7 usability dimensions of PACMAD in the smartphone-based physician-to-physician teleconsultation app, that is, completion rates, time to completion, number of clicks, number of screens, errors, incidents where the participant was unable to perform a task, which had previously been completed, and perceived task difficulty.

This study aimed to discover usability issues related to the smartphone-based physician-to-physician teleconsultation system, MEDIC, by using a mixed methods approach. From the summative evaluation perspective, MEDIC seems to be promisingly satisfactory as an app providing opportunities for efficient communication among health care providers, improvement of privacy protection, and increasing accessibility to patient records outside the clinic as well as in the context of long-term care. In addition, MEDIC is designed and perceived as a patient-centric platform where all related records and documents are organized for each patient. The participants generally compared MEDIC with their current practices in recording patients records and consulting with other health care providers (ie, paper chart, EMR system, and instant messengers). Using instant messengers can be frustrating when discussing multiple issues and cases in one channel. One participant reported that he normally writes patient records on paper and takes photos of the paper using his own smartphone. Although it can be convenient for capturing data, the quality of the photos of the records can be poor owing to the capturing process. In addition, it takes some time to retrieve the photos since such a process greatly relies on recall and memory. The current EMR system at the clinic is not portable and is not flexible in terms of serving specific needs and medical practice.

The results of participants’ performance (ie, number of screens used, number of clicks, time used) in relation to the experts’ performance can be considered as acceptable, considering that the majority of them had never used the app before. For memorability, the results yield a positive perspective since none of the participants forgot any actions that they had completed earlier during the test. However, usability issues were the most visible in learnability, errors, and cognitive load. Qualitative data were very helpful to explore users’ mental models in addressing these issues. Although these issues emerged from the interaction with MEDIC, most of the feedback can be applied to smartphone-based teleconsultation systems in general. The most frustrating task for some participants was creating a patient evaluation form (task 4) based on the number of clicks and time spent as well as the comments during and after completing the task. Although all of them were familiar with the assigned scale, that is, Mangled Extremity Severity Score, the participants who were not familiar with the vocabulary related to electronic form and questionnaire development (eg, dropdown, checkbox, select option) found it difficult to understand the interface for the first time. They took a longer time to complete it since they applied a trial-and-error approach to become familiar with the form creation process. Nevertheless, they commented that it would take them only a couple of hours to get familiar with this task. Creating a usable data collection form (eg, measurement scales, questionnaires) has been reportedly one of the most challenging tasks in system design and development from a broad perspective [42,43]. In a clinical context, numerous established measurements have been extensively used to assist the delivery of health care services. Creating a data collection form in a mobile app based on existing paper-based questionnaires can be challenging for novice users since it may require a different mental model [44]. Users may need to be familiar with the available features, icons, and labels to effectively create a form. To address this issue, an introductory guide or a tutorial video about form creation could be useful for users who are using this function for the first time (Multimedia Appendix 1). At the same time, further studies should be conducted to investigate the mental models of novice users particularly on creating an electronic data collection form in health care settings.

Another common error among participants was related to the limited image resolution of the smartphone. Neither computer screens nor mobile phones were initially designed to be medical devices. The display size affects the usability performance in multiple ways [45], particularly issues related to data presentation and input. In task 5, we used the image of a pelvic fracture with osteoporosis in a consultation task. It is noteworthy that 4 out of 5 orthopedists failed to recognize pelvic fracture on the screen, even though all participants increased the magnification of the image using the zoom-in function. After this revelation, they commented that they had not seen the fracture or had not paid attention to it because the image was too small. This may reflect the observation that the typical size of mobile phone display may not provide sufficient detail for a radiographic diagnosis [37,46]. In the acute management of multiple bone fractures, underlying conditions such as osteoporosis and other metabolic bone disease should be investigated preoperatively for proper surgical preparation and medical management to prevent unexpected complications. However, Hasselberg et al [15] found that diagnosis validity does not only depend on technology but also the users’ experience and physical ability. Certain solutions were suggested to mediate this issue, for instance, using an integrated DICOM viewer to provide better contrast, an ability to project onto a larger screen, and showing an image scale and other contextual information to raise awareness.

Another related issue as a consequence of screen size limitation is the misinterpretation and confusion about the image icons, which have also been addressed by other studies [7]. Owing to the limitation of screen size and the large amount of data, the design team decided to use image icons in place of text labels extensively in the app, especially for buttons. The decision led to issues related to naturalness, lack of information, and misleading information that are commonly found in other usability studies on smartphone-based health apps [37]. Participants who were using the app for the first time indicated that some of the icons were ambiguous and led to frustration. Some commented that some icons were too similar. When they were not sure what these icons were, they normally clicked to see where the buttons led to. This would cause frustration and be time consuming if the buttons did not lead to where they expected. However, some participants were successful in identifying the icons, referring to contextual elements, such as location of the icons, displaying content, and nearby icons. While replacing all icons with text labels would be immensely challenging, one possible solution is to use hover text, a tooltip text appearing when a user moves the cursor over a button. However, hover text is still not common in a number of developing platforms on touch screen devices. Other recommendations to improve the understandability of icons include removing unrelated or “unnecessary” icons, redesigning the icons to improve the distinctiveness among them, and providing a tutorial guide for the first-time user. These solutions are also suggested in other literature to avoid feature fatigue [47-49].

In addition to data presentation, the low resolution of smartphone screens can lead to data input errors [50-52]. We found that typing and clicking mistakes were omnipresent in all tasks. Participants commented that they usually had typing issues on their smartphones regardless of the app. They commented that the on-screen keyboard is too small. It is important to note that none of the participants used the swipe type function, where a user can glide his/her finger between characters. During task 3, one participant put the smartphone down and jotted down all the information on paper while talking to the simulated patient. He commented that typing on the mobile phone screen was difficult and required a lot of attention. Typing would significantly distract him from having a conversation with the patient. Alternative input methods were also suggested by the participants to remedy this issue, for example, adding an audio recording function and speech recognition ability as well as providing contextualized word suggestions. In addition, conditional formulae, such as deactivating a button/input when it is irrelevant, would help reduce errors in typing and other calculation tasks.

Drawing was also another input method recommended by the participants. Although MEDIC allows users to take photos, some participants commented that taking pictures alone might not be enough to capture all the information they would like to add. During the consultation in task 3, one participant mentioned that he wished to annotate the images taken by either drawing or typing next to the area of interest (eg, pain site). In addition, some of the gold-standard diagnostic scales require drawing as an input. For example, the Montreal Cognitive Assessment uses clock drawing to evaluate visuoconstruction skill [53]. Therefore, the next iteration of the app development has added a drawing function as well as image annotation as shown in Figure 4. However, it is important to further investigate the usability of drawing functions on smartphone screens since it has been reported elsewhere regarding user frustration [7].

In this study, the PACMAD framework was applied to guide the usability evaluation of a physician-to-physician teleconsultation app. We believe this is the first study utilizing this framework in this telemedicine context. PACMAD seems to be helpful to explore a broad range of usability issues of mobile apps in line with the heuristic evaluation approach. There is currently no specific usability framework specific to smartphone-based telemedicine apps. Although PACMAD was developed as an evaluation framework tailored to the usability of mobile apps in general [18], medical apps and systems can be more sensitive in certain usability aspects, for example, naturalness, consistency, and error prevention [54]. In addition, the evaluation in medical context should consider the complexity of usability from various perspectives, including user skills, task complexity, data sensitivity, and complicated functionality. Smelcer et al [23] argued that understanding the depth and breadth of user knowledge in context is central to the usability of EMR. There are specific frameworks/guidelines addressing the usability of medical systems, in particular [55,56], such as the TURF (task, user, representation, function) framework [25,40]. Applying these frameworks may have helped explore the complexity of usability issues in medical context; however, these frameworks were developed based on the context of desktop-oriented or web-based EMRs. Some are designed based on systems interacting with patients (eg, personal health records, physician-to-patient teleconsultation). It is apparent that PACMAD is more approachable and applicable to formative evaluation approaches where researchers are less restrained from sophisticated constructs and complicated research design. Even though applying more than one framework may be achievable, we found that both frameworks are not totally compatible. For instance, TURF includes both methodological and constructive guidelines, while PACMAD focuses more on usability dimensions. It would be ideal to develop a particular framework to evaluate the usability of medical apps on a mobile (and perhaps tablet) platform.

The main objective of this study was to explore the usability issues of MEDIC. Although Nielsen [39] suggests that 5 users would be sufficient to obtain the majority of issues, we found that a larger number of participants would enhance and increase the reliability of the results, particularly in the quantitative analysis. In addition, a more heterogeneous sample would be sufficient to investigate the variation of usability of EMRs based on different user characteristics found in other studies (for instance, between attending and resident physician [57]). In addition, although MEDIC is designed as a physician-to-physician teleconsultation app, the specifications and configurations of the app are still at the early stage. Furthermore, the tasks were developed in the context of an orthopedic clinic in a large medical school. Therefore, the results from this study may not be generalizable to a larger population. MEDIC is not designed to be used as a comprehensive standalone app but to be used in conjunction with other modes of communication among physicians (eg, instant messaging, EMR systems). The feasibility of an integration between patient-based teleconsultation apps and instant messengers should be further explored with respect to patient privacy and safety. Since it seems that screen size plays an essential role in the usability issues of smartphone apps, further studies should investigate alternative measures to prevent errors in data entry, which is the most visible and concerning issue found in this study. Although the display technology of smartphones has been progressively improved, other modes of data entry such as voice, drawing, and click-and-point may be considered in terms of feasibility and usability.

Since this study investigates the usability of smartphone-based teleconsultation in the early stage of the iterative design process, the purpose and approach of this usability study is rather exploratory than conclusive. While applying a mixed-method approach to gain a comprehensive perspective across all usability dimensions, based on PACMAD, we found that the qualitative data provided insightful perspectives and helped us discover usability issues in numerous aspects. In addition, since the goal of this study was to explore usability issues and the population was quite homogeneous, the feedback from 5 participants was sufficient to discover usability issues in all usability dimensions. Although there are a number of opportunities to improve communication among health care team members as well as between health care professionals and patients, we found that the usability issues of smartphone-based teleconsultation platforms in this study were mostly concerned with learnability, errors, and cognitive load. We found serious issues regarding errors particularly due to the limitation of screen size and resolution. Such limitations impact on how physicians enter and view patient’s records, which subsequently affect the diagnosis and treatment. Although the limitation of screen size has already been discussed in the literature, this study provides empirical evidence from a practical and user-oriented perspective. As in any early stage of development, there are still numerous opportunities for improvement, particularly regarding usability. An iterative process is planned to be adopted to develop the app to be more usable and expandable to a broader user group.