Telehealth is an umbrella term defined as “the provision of healthcare remotely by means of telecommunications technology,” whereas mobile health (mHealth) is an overlapping definition for “the use of mobile devices so that patients can solicit services electronically, use apps to verify information, and manage or monitor treatment or problems or other health-related issues” [1,2].

The exponential growth of telehealth and mHealth is revolutionizing health care delivery, because they have the potential to remove geographical barriers, increase access to medical services, and improve overall care quality [3]. Particularly, during the COVID-19 pandemic, there was a significant increase in the use of telehealth and mHealth, but the evaluation and its methodology have not matched the pace of its uptake [4,5]. Patients can thus receive support throughout the entire patient pathway, including app-supported diagnoses, therapy, and monitoring. Approved digital therapeutics (German: Digitale Gesundheitsanwendungen) are apps to improve treatment; their costs are covered by the statutory health insurance system in Germany. In the following discussion, we use “telehealth” to encompass the terms “mHealth” and “digital therapeutics.”

To achieve the greatest possible benefit from telehealth, usability is the key factor, especially with patients who have a cognitive limitation, are incapacitated by their disease, or are children [6-8]. This means that even evidence-based technology is not particularly effective for patient outcomes if it is difficult to use. This could be due to the technology itself or varying levels of eHealth literacy among users. According to Norman and Skinner [9], eHealth literacy is “the ability to seek, find, understand, and appraise health information from electronic sources and apply the knowledge gained to addressing or solving a health problem.”

Furthermore, measuring usability also protects patients from errors or experiencing harm. For example, if a certain telehealth system saves or displays medical data incorrectly and this leads to incorrect treatment, this inadequate usability can also disadvantage patients [10].

The International Organization for Standardization (ISO) norm 9241-11 defines usability as [11] “the extent to which a system, product or service can be used by specified users to achieve specific goals with effectiveness, efficiency and satisfaction in a specified context of use.”

Yet, many researchers have used additional attributes to assess usability [12]. A systematic literature study by Weichbroth showed that in descending order of priority, learnability, memorability, cognitive load, errors, simplicity, and ease of use were additional attributes used to assess usability in mobile settings [13]. However, this literature study excluded publications from medical and health subject areas. Sousa et al [14] conducted a systematic review of usability questionnaires for eHealth and showed that many existing usability questionnaires share these attributes but generally lacked effectiveness, cognitive load, simplicity, and ease of use. Interestingly, Sousa and Lopez [14] reported that the majority of usability questionnaires not only assessed usability but also the perceived usefulness in health care. The questionnaires included questions aiming to assess whether telehealth was helpful in fulfilling health care needs, which is not part of the usability definitions listed. The usability attributes aim only to assess the app’s efficiency, for example, “duration spent on each screen” or the app’s effectiveness, for example, “number of steps required to complete a task” [13].

On the one hand, authors of various studies used the single-item “Net Promoter Scale” (NPS) and the derived Net Promoter Score to evaluate telehealth [15-18]. However, the psychometric correlates of the Net Promoter Score are not clear, and it is also thought to measure satisfaction and acceptance [15]. We believe that the NPS seems to be a valuable instrument because it offers a straightforward, quantifiable, and very short measure of the user experience and is easy for patients to understand and respond to accurately. Its numerical scale facilitates clear aggregation and analysis of data, allowing for effective comparison over time and across patient groups. The categorization into promoters, passives, and detractors appears to provide actionable insights for further improvements and its widespread use across industries, including health care [19].

On the other hand, up to 38-item questionnaires were used to measure usability [20]. We have, therefore, decided to include the NPS in our study to assess its association with known usability attributes.

A closer look at usability questionnaires reveals a need for development. First, different questionnaires exist side by side, sometimes measuring only different facets of usability or usability-related constructs [14]. Second, many questionnaires have little empirical evidence regarding their psychometric properties or, third, they can only assess the usability of a specific or single technology [14]. Fourth, most of the questionnaires are only available in English. Accordingly, there are hardly any validated and appropriate questionnaires for usability studies in the German language [21]. As far as we know, only the German translation of the System Usability Scale—the origin of all usability questionnaires—seems to be available for wider use so far [22]. However, none of the 4 existing German versions is convincing [23]. The first 3 points also become clear when you look at the small number of questionnaires available in German.

Some questionnaires are available in German capture usability-related constructs, but do not focus directly on usability (eg, Mobile App Rating Scale-German [MARS-G] [24] and the User Experience Questionnaire [25]). The AttrakDiff questionnaire measures usability as merely 1 dimension among several others [26]. More specifically for the telehealth area, Altmann et al [27] published a German version and a short version of the “Telemedicine Perception Questionnaire” (TMPQ) in 2022. The original questionnaire includes 17 items designed to evaluate the patients’ impressions of home telecare, as well as to assess its potential risks and benefits. Thus, the TMPQ does not measure usability per se (see first point) and is limited to evaluating older patients receiving video consultations from a nurse (see third point). Moreover, the validation of the questionnaire is limited to only 32 and 10 participants in its validation study [28] (see second point). The German translation could be shortened after subgroup analysis to a short version with 5 items. The German version of the TMPQ showed sufficient reliability (Cronbach α=0.76) in Altmann’s study with 32 participants compared to the original study (Cronbach α=0.8). For the brief version, reliability was still acceptable with Cronbach α of 0.72.

There is another small number of questionnaires available in German whose area of application is very limited (eg, ISONORM 9241/110 on desktop apps [29]). Specific to the telehealth area, different authors in this field offered German translations of the mHealth App Usability Questionnaire (MAUQ) [30-32] (see third point). Moorthy et al [30] validated their translated MAUQ in a specific sample of 133 patients with cancer but, presumably due to the small sample size, the factor structure of the translated questionnaire was not further investigated. Kopka et al [31] provided a German version and a German short version in a sample of 148 patients using a symptom checker app in an emergency department in a randomized controlled trial. They showed that the original factor structure did not fit the data well, but no further investigation of the factor structure was conducted. In their validation study (n=53; see second point), Tacke et al [32] showed a strong positive correlation between their MAUQ translation and the System Usability Scale (SUS). However, the factor structure was not examined due to the sample size.

Considering these shortcomings, we still see a need for an appropriate questionnaire available in German. Our aim is to ensure that this German questionnaire is suitable for assessing the general usability of both telehealth and video consultations. In addition, the factor structure is to be evaluated on the basis of a sufficiently large dataset.

The Telehealth Usability Questionnaire (TUQ) by Parmanto et al [33] in 2016 measures all usability attributes except memorability, allows the evaluation of video consultations, and is the most used usability questionnaire [34,35]. The TUQ uses preexisting items from other questionnaires and is freely available following the Creative Commons license 4.0. The TUQ is recommended by other authors and by frameworks for assessing telehealth [14,17,33-35]. We, therefore, translated and validated the TUQ in German (see Methods for further information on the TUQ).

Bibiloni et al [36] published an exploratory factor analysis (EFA; 150 questionnaires) of the TUQ [33] relating to video consultation. They found that 2 factors were sufficient to model the observed data. After the questionnaire was adapted to 12 items by an expert team, a confirmatory factor analysis (269 questionnaires) was performed. Both factors could be measured with good reliability, but they were highly positively correlated. Despite adapting the items and shortening the questionnaire to 12 items, the main problem with the questionnaire was that no good differentiation between usability and perceived usefulness in health care could be achieved. Although the high factor correlation raised the question of whether respondents differentiate between these 2 aspects, a 1-factor model showed a clearly worse fit than the 2-factor model. A limit to the application of the short questionnaire of Bibiloni et al [36] was that the inclusion criteria merely required a single instance of a video consultation and thus did not allow for the general evaluation of apps in telehealth. Besides, as the factor analysis of Bibiloni et al [36] confirmed, the TUQ measures usability, as well as perceived usefulness in health care, and the name “Telehealth Usability and Usefulness Questionnaire” would be more appropriate.

The primary objective of this study was to develop and validate a German language version of the TUQ and compare it to the NPS. This adaptation aims to make the TUQ readily accessible and broadly applicable for evaluating telehealth usability within German-speaking populations. The second aim was to reduce the number of items in order to optimize the feasibility for use in general field studies in telehealth.

Stage I (April 2022-January 2023) consists of translation, adaptation, and pilot-testing of face validity as a method of construct validity. Stage II (July 2022-September 2023) consists of the development of a short-scale—psychometric testing and final item selection. The reporting of this study has been structured according to the recommendations of Streiner and Kottner [37] for reporting the results of studies of instrument and scale development and testing.

High-quality translations can only be produced if measurement instruments are linguistically replicated and culturally adapted as rigorously as possible. Sousa and Rojjanasrirat [38] presented a guideline on “Translation, adaptation and validation of instruments or scales for use in cross-cultural health care research.” In agreement with the lead author, Bambang Parmanto, we (MD and JZ) conducted translation and cross-cultural adaptation with a multidisciplinary expert committee. The members of the expert committee were selected on the basis of their knowledge and experience in the area of application of the questionnaire and their language skills, as reflected in their previous research activities and clinical experience. Translations were carried out by native speakers with proven expertise in the field of application and a high level of language skills in the target language of the translation. The recruitment took place within the authors and through the authors’ network.

To include users’ opinions and views on the German version of the TUQ, we conducted pilot-testing with medical staff, as well as a relative of palliative care children (Figure 1). Several staff members with a high level of clinical experience and patients in palliative care and their relatives were asked to participate in the pilot-testing. The aim of the test was to check whether the items were clear, understandable, and comprehensible. We recruited via the network of authors. The 3 nurses, 1 physician, and 1 relative who participated in the test were asked in an interview to indicate whether the item was clear or not (clear or unclear) and to briefly formulate how they understood the item (thinking aloud method). As a result, we made minor linguistic adjustments.

The study was approved by the institutional review board of the Medical Faculty of the University of Erlangen-Nuremberg, Germany (22-425-Bm; January 25, 2023); the University of Kassel, Germany (202213; April 28, 2022); and University of Giessen (AZ 64/22; September 16, 2022). This study is registered in the German Clinical Trials Register (DRKS00030546). Participation in the survey was voluntary. All patients gave their written informed consent before study inclusion. All patients who participated in this study were coded with a consecutive number in a pseudonymization procedure. The data collected were stored and analyzed in a password-protected database. Only previously defined and authorized persons had access to this data. Patients had the option of withdrawing their participation in the study at any time, whereby all personal data were irrevocably deleted. The study was conducted in accordance with the ethical guidelines of the Declaration of Helsinki.

The TUQ was translated in a step-by-step protocol shown in Figure 1. The expert committee discussed several minor cultural and linguistic differences, and the original developer approved all the adjustments. Original TUQ item 4 “It was simple to use this system” was swapped with item 5 “It was easy to learn to use the system,” as multiple pilot user testers were irritated by this pair of questions.

The TUQ questionnaire is designed to assess different types of telehealth and, depending on which technological application is to be assessed, its wording can be adapted accordingly. There was a need to clarify the terms “telehealth,” “system,” and “telehealth system,” which were all replaced by the term “app” to make the questionnaire applicable to apps. See Table 1 for the complete translation.

The complete TUQ was translated into German and cross-culturally adapted. It is comprehensible and equivalent to the English version [33] (see Table 1).

We identified 2 factors (“perceived usefulness in health care” and “usability”) in the TUQ which are sufficient to describe the patient’s perception. The NPS does not allow an assessment of usability attributes. The TUQ could be shortened to 6 items without compromising its informativeness as discussed.

The TUQ shows a 2-factor structure—on the one hand “usability,” on the other hand “perceived usefulness in health care.” Both factors correlate highly positively (r=0.59, 95% CI 0.56-0.63). The same factors were also shown as the main factors of the Spanish version of the TUQ by the working group of Bibiloni et al [36] in 2020. A factor analysis of the Thai version of the TUQ also resulted in a 2-factor model, comparable both to our study and Bibiloni et al [36,55]. The 2 factors were “accessibility” and “utility.” To the best of our knowledge, no further data on the TUQ factor structure were found in other studies [34]. In line with this research, and despite the high correlation of the factors, we deemed the 2-factor solution plausible for 2 reasons—first, it is conceivable that a health care app is technically well-designed but does not fulfill its health-related purpose for the patients. Hence, these aspects are distinguishable, and a 2D questionnaire encourages respondents to think about these aspects separately. In that sense, the high correlation may be an artifact of the context in which patients with highly specialized health care apps were asked to fill in the questionnaire. Second, previous research [56] has shown that factor analysis tends to err in the direction of extracting too few rather than too many factors, thus, the results underline the theoretical notion that there are 2 distinguishable factors in the TUQ.

We removed TUQ items (see Table 1; items 2, 5, 6, 8, 17, 19, and 21) largely due to high cross-loadings in order to sharpen the interpretation, arriving at a short version with 3 items per factor (see Multimedia Appendix 8). As the TUQ, similar to many other usability questionnaires, also contains items measuring the perceived usefulness, we, therefore, propose a clear title stating the dual purpose of the short questionnaire, thus “Telehealth Usability and Perceived Usefulness Short Questionnaire for patients” (TUUSQ). Bibiloni et al [36] proposed a short version with 12 items to assess usability in telehealth focusing on video consultations. This short version shares 4 items with the TUUSQ (see Table 3).

The factor that we named “perceived usefulness in health care” was included in the TUUSQ with 2 items relating to health care access and support. In addition, 1 item regarding the intention of future use (Cronbach α=0.79) loads this factor. We interpret this finding as the future intention of use is highly associated with perceived usefulness in health care and not with good usability experience. Other studies in other contexts also demonstrated that satisfaction and future intention of use show a high correlation [60-63].

The second factor, “usability,” contains 1 item each regarding ease of use, reliability, and interface quality. The TUUSQ thus lacks the TUQ items addressing the usability attributes as defined by Nielsen [12], that is, “efficiency” (item 6), “efficacy” (item 2), and “satisfaction” (item 21). However, as our data show no benefit in adding further items, and that the feasibility of a 6-item questionnaire is very good; therefore, we advocate this short version. Interestingly, in contrast to the usability attributes, the TUQ item addressing interface quality showed the second-highest factor loading for the factor usability in our study. One of the first usability questionnaires, the “Post Study System Usability Questionnaire” (PSSUQ) showed a 3-factor model with the factors “interface quality,” “system usefulness,” and “information quality” [57]. Saeed et al [64] also showed that, in the context of telehealth home monitoring, the quality of the user interface is of utmost importance for patient usability. Weichbroth [13] reported that among others less commonly assessed usability attributes for the mobile setting include navigation, operability, attractiveness, aesthetics, accessibility, and interaction [13]. Possibly our German translation of the TUQ item 7 “The way I interact with this app is pleasant” which means literally “the app’s user interface has a pleasant design” also relates to these attributes (see Table 1). This finding was also present in usability studies in other health care settings [65].

The original TUQ contains no items regarding memorability. We decided against adding an item to assess memorability as our study design as it does not allow testing for memorability, that is, ease of reusability of the applications after long periods of disuse. Of course, the usefulness of telehealth should be memorable and practical in the longer term, but this question was not part of this study. Further studies on this with a more extended use of telehealth should be carried out in the future.

Interestingly, the NPS did not show moderate to strong correlations with either the 2 identified factors—perceived usefulness in health care and NPS: 0.105 (95% CI –0.00 to 0.21); usability with NPS: –0.11 (95% CI –0.22 to –0.01) or with any of the individual TUQ items. These results show that the underlying construct of NPS is not associated with any of the patient’s usability or perceived usefulness in health care attributes covered by the TUQ in our studies. As the TUQ contains an item to assess satisfaction (item 21), our studies support the findings of Krol et al [15] that the NPS is not associated with patient satisfaction. The construct that is measured by the NPS in health care continues to remain elusive [66]. Future research assessing health care quality should include the NPS in conjunction with larger surveys [66]. This is necessary since single-item measures are notoriously unreliable—potentially explaining this null finding—and should not be used as a critical variable in high-stakes settings. Moreover, possibly a correlation to other attributes might reveal the underlying construct of the NPS and pave the way to evidence-based use in health care. Finally, it should be noted that Adams et al [66] suggest limiting the use of the NPS to certain health care settings, for example, where patients have a choice of provider. This was not the case in our study.

The items selected from the TUQ for the TUUSQ originate from the PSSUQ and Telehealth Satisfaction Questionnaire (TSQ) questionnaires (see Table 3 [57,58]). All TUUSQ items addressing usability originate from the PSSUQ. The PSSUQ was reviewed by Sousa and Lopez [14] in 2017 and assessed as one of the best available usability questionnaires, although the TUQ was not included in this study [33]. The PSSUQ’s items were generated using an empirical study and showed very good internal consistency (Cronbach α=0.97). However, the quality assessment of validity, reliability, user-centeredness, sample size, and feasibility by Sousa and Rojjansrirat [38] yielded a medium-quality score due to low sample size and lack of reported user-centeredness during item generation. The PSSUQ is also sensitive to user-group and system differences [57].

The TUUSQ items assessing perceived usefulness in health care all originate from the TSQ. The TSQ shows a 3-factor model [58] and all items used by the TUQ and thus TUUSQ belong to the factor called “quality of care provided.” No review of the psychometric properties of the TSQ is available. The reported sample size was low and lacked reported user-centeredness but showed good internal consistency (Cronbach α=0.93).

The TUQ was translated in a way that has proven itself in research. Nevertheless, individual words may seem inappropriate for some target groups (eg, item 15—“Hausbesuche” for “in-person visits”). Depending on the context, “Vor-Ort-Termine,” for example, may seem more appropriate here. We would like to point out that such adjustments can have an impact on the quality of the questionnaire.

The development of the TUUSQ short questionnaire was for the main part only examined patients who received digital therapeutics or used a telehealth system. Further studies are needed to evaluate whether the short questionnaire can be successfully applied more widely in the area of telehealth. Furthermore, the applicability of the TUUSQ may be limited in other cultural contexts, as this study only included patients from Germany. This limitation also applies to the original instruments (PSSUQ and TSQ) used as references. Cultural and contextual differences between the original settings of these instruments and this study could affect their relevance and accuracy in different populations. Thus, we encourage further studies using our translated version to analyze the factor structure in other datasets to investigate the generalizability of our conclusions.

The TUUSQ offers a short and highly feasible questionnaire for assessing and distinguishing the perceived usefulness and usability of telehealth. The TUUSQ contains solely generalizable items which allow for its use in many different telehealth contexts, as advocated by Sousa and Lopez [14].

Based on our results and the binary factor TUUSQ structure and the recommendations of Sousa and Rojjansrirat [38], we propose the following sequential approach for the assessment of telehealth apps—first, assess whether the app addresses a relevant health care need for patients. If not, and patients report no improved access to health care and health care support, or future intention of use is not reported, the scope of the app should be reevaluated. If perceived usefulness in health care is given, the TUUSQ provides software developers with concrete information on the app’s usability giving feedback on ease of use, reliability, and the quality of the user interface.

Following our study, the TUUSQ can be used in Danish, German, Portuguese, Slovene, Thai, and Urdu, as validated TUQ translations are available for these languages [33,55,67,68]. The TUUSQ is free for commercial and noncommercial use following the Creative Commons license 4.0 [69]. If, besides perceived usefulness in health care and usability, the quality of the video connection during a video consultation, as well as the suitability of this medium are of interest to the researchers, we recommend using the TUQ short version as proposed by Bibiloni et al [36].