This is a mixed methods study using (1) the System Usability Scale (SUS), (2) the Net Promoter Score (NPS), and (3) a thematic analysis based on participant’ perceptions. The aim was to assess the usability and favorability of health care provided in a connected cabin for symptomatic and asymptomatic employees of the BP Hospital (Beneficência Portuguesa de São Paulo).

Ethics approval (CAAE: 58070622.9.0000.5483) for this study was provided by the institutional review board of the BP Hospital (Beneficência Portuguesa de São Paulo) on January 31, 2023. All eligible patients signed an informed consent form for this research protocol.

The data was collected on the RedCap platform, which has access control to ensure the security of the data collected in the research. In addition, all data was anonymized to ensure the privacy of the participants.

The Binomial test was used to calculate the sample size for both parts of the study. For the sampling of asymptomatic individuals (phase 1), an 80% test power was stipulated, a 70% acceptance rate of the cabin under the null hypothesis, and a 90% expected in the sampling rate were specified, resulting in 41 asymptomatic patients.

For the sampling of symptomatic individuals (phase 2), we opted for a conservative scenario. Thus, we set a power of 90%, an acceptance rate of 80% under a null hypothesis, and a 90% expected proportion in the sampling, resulting in 169 patients. Anticipating the possibility of a 12% sample loss, the researchers decided to increase the sampling to 190 symptomatic patients.

The telemedicine cabin Diagnostica, manufactured in Argentina, was used. It is made of plastic and fiberglass and measures 250×150×230cm dimensions. It has a forced air ventilation and filtering system, environmental lighting with intensity and color control, and an audio and video system for videoconferences (Figure 1).

The participant used his cell phone or the device available in the cabin to receive care and to interact with the person through the app developed for Android and IOS. The patient was accommodated in the cabin and selected the self-examination devices according to their needs.

The following devices were available in the cabin: an otoscope, a digital dermatoscope, and a high-definition camera, all manufactured by Firefly Global in 2021; a stethoscope Riester manufactured in 2021; and an oxyhemoglobin saturation monitor (oximeter), an electrocardiography, a heart rate and temperature monitor, and a sphygmomanometer, models PM6100, all manufactured by Berry, 2021. The patient was instructed to use the app on his cell phone, and the light signals in the cabin were added to a video display to operate the devices for the self-examination.

The cabin recorded the measurements taken by the available devices. The patient had the option, at their discretion, to join the medical consultation at any time during the use of the cabin, even if he did not use the devices. During the synchronous service, the doctor could request that the patient use any of the medical devices to obtain additional clinical information.

Doctors were recruited through direct invitations sent to professionals associated with the hospital. Selection criteria included previous experience with similar technologies and availability to participate in all stages of the study.

Data between the patient and the doctor was transmitted via the Doc24 telehealth platform, implanted in the data processing centers, and connected to the internal service network.

Before starting the service, a nursing technician applied the informed consent form. After the participant’s agreement, a screening questionnaire focusing on the inclusion and exclusion criteria and a questionnaire to collect demographic and socioeconomic data before using the cabin were administered. Individuals who met the exclusion criteria or had acute alterations requiring face-to-face assessment were sent to the Employee Health Center for immediate medical care.

The cabin was equipped with a Wi-Fi network and the patient should download an app to access the cabin and the interface (application) which was customized for the project.

During the appointment, the patient, seated in the cabin, can choose between selecting devices for self-examination or opting for a consultation with a doctor. At the patient’s discretion, they have the option to join the medical consultation at any time during the use of the cabin, even without having used the devices.

If the participant chooses to attend a medical consultation during the service, the doctor may ask the patient to use some of the equipment to collect clinical information, or to reuse some of the equipment due to observing poor quality results initially measured by the patient.

Equipment-assisted measurements were available for medical assessment, if required, or later, in the outpatient treatment.

At the end of the service, the research participant was invited to answer the usability questionnaires translated and validated including the SUS, which is a method to measure the usability of several products and services through 10 questions answered on a scale of 1 to 5, where 1 is total disagreement and 5 is total agreement [9], and the NPS, which aims to analyze the patient’s experience and satisfaction [10].

After assigning the NPS score, participants were asked to explain their ratings. Inductive thematic analysis was used to identify a rating [11]. Two authors carefully read each response individually and grouped the responses. Any disagreement was discussed with the first author, and then the response was labeled into 6 categories: “service,” “usability,” “difficulty of use,” “innovation,” “technology resistance,” and “unspecific.” The final step was to group the responses into potential themes.

In order to check the usability of health professionals, the participating doctors also completed the same questionnaires at the end of the daily sessions.

Study data were collected and managed by an electronic data capture tool, REDCap (Research Electronic Data Capture), and hosted on the BP server [12,13]. REDCap is a secure software web-based platform designed to support data collection for research studies, providing (1) an intuitive interface for the validated data collection, (2) audit trails to track data manipulation and export procedures, (3) automated export procedures for further data downloads to standard statistical packages, and (4) procedures for data integration and interoperability with external sources. Once collected, data were described considering the mean and SD for the numerical variables and the absolute and relative frequencies to the categorical variables. SUS scores were interpreted using the Sauro and Lewis Curved Grading Scale [14].

Patients were classified into 2 groups according to the SUS scale score “Acceptable usability” (SUS ≥68) and “Usability issues” (SUS <68), and the percentage of patients was compared with 70% (for the Symptomatic sample), and 80% (for the Symptomatic sample) by the Binomial test. The cut-off point of 68 was selected because, despite variability in acceptability, this is the threshold at which the technology is considered to have acceptable usability.

The chi-square test or Fisher exact test was used to compare the groups’ qualitative characteristics, and the Mann-Whitney Test was used for the numeric characteristics. The significance level of 0.05 was used throughout the study, and the analyses were performed using SPSS software (version 25; IBM corporation). Graphs were generated using R (version 4.3.2; R studio Team) software and the ggplot2 package.

The study was divided into 2 phases: the first occurred from March 6, 2023, to March 16, 2023, and 40 asymptomatic patients were included. The second occurred from March 17, 2023, to June 1, 2023, and 189 symptomatic patients were included.

The demographic distribution of the asymptomatic patients is shown in Table 1. Most were female, White, married, university-educated, and of higher socioeconomic status.

Concerning the use of the cabin, most of the patients requested support to use the cabin at some point during use. Only 3 patients described the reasons for requesting assistance as difficulties in using the equipment. The appointment had a duration between 10 and 52 minutes.

Upon assessing the satisfaction (Figure 2), we observed that the SUS had a mean value corresponding to Grading A+ (Table 2). When defining the cut-off point for SUS, we identified that only 5 (12.5%) patients had a score of <68 points, whereas 35 (87.5%) had a score of ≥68 points. Thus, the acceptability of 87.5% was statistically different from 70% (P=.009) and had a 95% CI 75.5%-100%, justifying the acceptability of the asymptomatic sample and allowing the study to proceed with the sampling of symptomatic patients.

The demographic distribution of the symptomatic patients is shown in Table 2. Most were female, white, single, had a college degree, had a household income up to 3 minimum wages, did not perform care functions, and had a mean age was 35.1 years.

Only 5 (2.7%) patients requested support to use the cabin outside the previously established times. The 3 reasons described for requesting support were inconsistencies in performing the cabin and difficulties in using the instruments (otoscope and oximeter) available in the cabin.

The most of participants requested medical orientation and performed the measurement under medical orientation. The duration of the appointment ranged from 2 to 58 minutes. Regarding satisfaction (Figure 3), the variable SUS mean corresponds to Grading A+ (Table 2). When assessing the participants’ score in relation to the cut-off point of the SUS scale, we found that only 21 (11.1%) patients had scored <68 points, while 167 (88.8%) had scored ≥68 points. Thus, 88.8% acceptability was statistically different from 80% (P=.001) and had a 95% CI 84.3%-100%.

The 21 patients with a score of 68 or less on the SUS were considered in the group “Usability issues” at the cabin. Eleven (52.4%) were female, 13 (61.9%) self-declared White, 11 (52.4%) had a college degree, 13 (61.9%) were single, 5 (45.5%) were married, 7 (63.6%) earned up to 3 minimum wages, 8 (72.7%) did not carry out welfare activities, and had a mean age of 32.64 years (SD 8.43).

Regarding the use of the cabin by these participants, none of them required assistance from the nursing technician, 8 (38.1%) participants requested medical instruction to perform the tests, 7 (33.3%) participants requested medical orientation and made the tests under medical orientation, while 6 (28.6%) participants did not request medical orientation.

The mean appointment duration was 20.38 (SD 10.29) minutes, ranging from 6 to 40 minutes. When comparing the characteristics of patients with SUS ≥68 related to SUS <68, all of them were not statistically significant (attaining a value equal to or higher than P=.05) (Table 3), meaning that we did not identify characteristics that could have influenced the difference in the SUS score.

Regarding the reasons for their ratings, only 3 participants chose not to answer this question. After careful reading of each response by 2 researchers, the answers were grouped according to the following categories:

The distributions of the classifications are shown in Table 4.

Most opinions are classified as related to “Usability.” When comparing the classes of sentences of those who scored SUS equal or less than 68 (usability issues) with the remaining (acceptable usability), we found statistical differences in the class “Difficult Usability.” In the group “Usability issues,” 5 (27.8%) scored in this class, while 8 (4.8%) scored in the group “Acceptable usability.”

At the end of the day’s work, the doctor performing the appointments was oriented to answer the NPS and SUS, evaluating his or her day’s work. The assessment was performed by 2 doctors, reducing the variability of the influence of the medical service on usability; 89% of the services were performed by one of the doctors, and 11% were performed by the other. The distribution of the NPS and SUS is shown in Table 5.

The main difficulties were related to technical difficulties, such as software updates and network connection problems, besides difficulties in adjusting the cabin’s technology (problems with the sensors for measuring the weight and height; problems with the image or screen in the cabin).

Figures 4 and 5 show the evolution of the allocation of medical grades over time.

The assessment of the implementation of the telehealth cabin in this study aimed to observe the level of favorability and usability of users (doctors and patients) to the cabin and the self-examination devices available, added to their experience of remote service through the technologies available.

The analysis of the results showed that the participants of the first phase of the research (asymptomatic patients) had a different socioeconomic profile than those of the second phase (symptomatic patients), with higher education and monthly income, probably due to the new technology, which attracted leaders and directors of the institution to the research. Although this group was only used to assess the acceptability of both the cabin and the self-examination devices in order to continue the study with symptomatic patients, the outcomes of the favorability and usability were similar between the groups.

Therefore, despite the disparity in access to technology among populations of different economic levels, such a result suggests that the telehealth cabin may have broad applicability for several publics due to the increasing availability and use of mobile devices among the whole population, thus promoting health the equity and reducing social inequalities.

A study with similar equipment performed with French students also concluded that the personal relationship with the technology did not influence the intended use of the telemedicine cabin [15].

A significant portion of the participants did not work in the hospital care sector and therefore did not routinely use the self-examination devices, suggesting that lack of familiarity with the use of the self-examination devices may not impact the use of the cabin. However, this study was performed with professionals working in a hospital, which may bring a certain level of familiarity with those devices, even for those not directly working in the care sector.

Still, we observed that more than half of the patients requested an interview with the doctor, either before or after self-examination, indicating the relevant role of the doctor in such a kind of service.

High favorability and usability were observed for both patient groups (symptomatic and asymptomatic). No statistically significant differences were found when comparing the socioeconomic and cultural profile of those patients with the group of symptomatic who considered the cabin to have acceptable usability, which shows that the usability and favorability of the cabin are unrelated to socioeconomic and cultural factors. However, concerning comments from this group show that some aspects were considered positive even in this group. Furthermore, only one patient described a comment that could be classified in the category “Resistance to Technology,” suggesting that most of the population tested is interested in learning and adopting technologies that improve their routines and experiences.

The findings of this study support the work by Scott Kruse et al [8], asserting that the principal barriers to telemedicine are technology specific. They can be overcome by continuous improvement of these technologies, adequate training of technology users, and personal interaction between the patient and the care provider [8].

Concerning the favorability and usability of doctors who performed the cabin services cabin, we found that the answer was also positive, but with a slightly lower level of usability than that of the patients. This fact seems to agree with other studies that assert that some health professionals may be reluctant. However, this resistance may be related to the difficulty of the technology. Therefore, both the favorability and usability can improve with the advancement of technology and the due training of doctors in web-based physical tests [16,17]. This phenomenon highlights the importance of training programs as a strategy to improve user experience and expedite the adoption of this technology.

A limitation of this study is the participation of only individuals without respiratory symptoms (a consequence of the COVID pandemic) or without symptoms that could indicate a medical emergency. Another limiting factor is the poor accessibility of the cabin for people with disabilities. Furthermore, because the total sampling included people employed at the hospital, and so, in the productive age group and predominantly female [18], along with workers in the day shift, this study may have introduced a bias in the perception of favorability and usability of the participants (for instance, older men are less prone to participate in several telehealth activities [19]).

Finally, we emphasize that the results obtained from health professionals should also be interpreted with caution due to the limited number of professionals who provided care for the study.

It should also be noted that no studies using this type of technology were identified in South America to compare the results obtained. In addition, the fact that this study was conducted after the pandemic may have influenced the results given the impact of the COVID-19 pandemic on people’s lives, which may have affected perceptions.

Based on the favorable assessment of the technology by the participants in this study (patients and doctors) new studies that extend its use to different populations, without any link to the hospital and from different sectors of activity, different work shifts, sex, and different age groups, may bring new inputs to the application of such technology, expanding the service types and favoring the inclusion of the service in remote locations and for diverse populations.

The authors conclude that the telehealth cabin had good usability and favorability by patients, regardless of the socioeconomic and cultural profile of the population. Doctors who performed the cabin services attested to the cabin’s good usability cabin but with a usability index slightly lower than patients. Most patients used the devices for self-examination, with little need for technical assistance. However, a SUS score greater than 68 does not necessarily mean that there are no usability issues to be addressed.

The outcomes of this research allow us to assess as feasible the expansion of this kind of practice from the point of view of usability and that this study can contribute as a subsidy to the construction of public policies to expand the access of the Brazilian population to qualified medical care, promoting the engagement of patients in their health, providing health professionals with an extended view of the clinical conditions through embedded devices even in areas with a shortage of this type of assistance.