Since the development of ELIZA in 1966 [1], a rule-based system developed to emulate text-based interactions with a psychotherapist, advancements in information technology have driven the widespread adoption of health care conversational agents (HCAs) [2,3], which are applications designed to support interactive, multiturn communication through written or spoken language [4]. Consumer solutions such as Siri, Google Now, Cortana, and Alexa are well-known examples of agents designed to help users find information and accomplish tasks [5].

Modern HCAs extend well beyond simple rule-based input-output interactions. Advances in large language models have facilitated their evolution from narrowly defined, task-oriented tools into open-domain systems capable of retaining conversational context, following topics across multiple turns, and sustaining coherent, continuous dialogue for more natural interactions [6,7]. In addition, AI techniques have enabled more advanced dialogue management, facilitating the exhibition of human-like behaviors, the recognition of users’ emotions and intentions, and the use of various information sources for reasoning and decision-making [8-10].

Building on these advancements, researchers have explored anthropomorphized interfaces, leading to the development of embodied agents, or virtual humans [11,12]. Unlike traditional agents, which rely solely on text or voice, embodied agents combine verbal communication with nonverbal cues (eg, posture, gestures, eye movements, or facial expressions) to foster more lifelike interactions [13-15]. This embodiment enhances users’ perception of social presence and engagement, making interactions feel more intuitive and emotionally rich [16,17]. Moreover, the introduction of virtual reality technologies has paved the way for increasingly sophisticated agents designed to promote user engagement through immersive experiences [18].

HCAs are widely used in health care [19], reflecting the widespread adoption of similar agents across other human-centered sectors, including customer service [20], entertainment [21], and education [22].

From a health care research perspective, the assessment of HCAs should consider health-related outcomes, such as symptom reduction, disease knowledge, disease self-management, and quality of life [8]. This assessment should be conducted in the context of continuous use of the applications and preferably supported by randomized controlled trials. However, before conducting randomized controlled trials, it is important to safeguard aspects that might affect health-related outcomes. This means that preliminary assessments (ie, feasibility studies) should be conducted [23] to evaluate the technical characteristics of HCAs or their quality in terms of usability and user experience and, consequently, their potential to promote user engagement among the target users [8,24].

System usability comprises several components (ie, learnability, efficiency, memorability, errors, and satisfaction) and relates to the ease of using the system interface [25]. The rise of new technologies, devices, and interaction methods has expanded the concept of usability into the broader notion of user experience, which encompasses the cognitive, affective, social, and physical aspects of interaction [26-28].

There are 3 main methods for evaluating usability and user experience, namely, inspection, testing, and inquiry [29]. The inspection method consists of a systematic analysis of interaction mechanisms carried out by experts according to specific guidelines, while the testing and inquiry methods are based on the collection of usage data [29]. The testing method involves observing users while they perform tasks with a given system to collect primarily quantitative data (eg, the number of completed tasks) that can provide empirical evidence on how to improve the interaction mechanisms [29]. By contrast, the inquiry method involves collecting qualitative data from users, namely through interviews or subjective measurement instruments such as questionnaires or scales [29]. Examples of commonly used instruments include the Post-Study System Usability Questionnaire (PSSUQ) [30] and the System Usability Scale (SUS) [31]. These instruments have had their measurement properties assessed, particularly validity (ie, how effectively usability or user experience is measured) and reliability (ie, the consistency of measurements when repeated under comparable conditions) [30-33].

In this context, this review aims to synthesize the current evidence on usability and user experience assessments of HCAs using valid and reliable subjective instruments, to identify the purposes and categories of HCAs being assessed, and to determine, through a comparative analysis, how this typification affects their usability and user experience.

Several reviews [34-41] have already focused on different aspects of HCAs (eg, the use of AI or large language models), including usability and user experience assessments (eg, [42-45]); however, these reviews did not perform comparative analyses of usability and user experience across different categories of HCAs. Such analyses are important because different characteristics of HCAs affect how users interact with them (eg, by shaping communication style, pacing, and information processing), as well as cognitive load and perceptions of trust and privacy. Therefore, this review may provide insights to inform better design decisions and the development of standardized evaluation frameworks, enabling researchers and practitioners to benchmark usability and user experience across different HCAs.

This review provides an overview of the current evidence on the usability and user experience of HCAs assessed using validated subjective instruments. As HCAs become increasingly widespread, a growing number of individuals interact with them daily, and it is anticipated that these agents will play an important role in future interactive systems. Therefore, it is crucial to investigate the assessment of their usability and user experience, particularly in the context of emerging human-machine interaction, because well-designed assessments contribute to the overall improvement of interactions. Although well-designed interactions do not necessarily mean that digital health applications are effective, their efficacy and effectiveness may be limited by usability and user experience constraints [215].

Patient care emerged as the predominant application domain for HCAs, particularly in mental health. Other reviews [8,216] have already identified the importance of patient care, while the efficacy of HCAs in supporting mental health disorders has been the subject of several systematic reviews (eg, [217,218]) and a meta-analysis (eg, [219]). Among the other application domains addressed, HCAs may support health education and prevention by optimizing the delivery of understandable, personalized health education at scale and by providing interactive and engaging information across platforms (eg, mobile, web, or smart devices) [220,221]. Similarly, health data collection has already been identified as a core use case of HCAs [216,222], namely, in terms of automating diagnostic interviews and longitudinal monitoring of patient-reported outcomes (eg, anxiety or depression scales) to track symptoms and adherence to treatments over time. Furthermore, the use of HCAs designed to promote the independence and autonomy of older adults by assisting them in managing activities of daily living and self-management of health conditions is in line with current interest in developing digital applications to support the care and well-being of older adults [223,224]. Another relevant point is that a high proportion of the studies included older adults, who accounted for 875 of the 7040 (12.43%) participants.

Although text-based HCAs were the most frequently evaluated, all 3 identified categories were applied across different health care domains. This suggests that developers select interaction modalities according to technological feasibility and user requirements rather than the clinical purpose itself.

The increasing integration of virtual reality with embodied HCAs reflects growing interest in immersive human-computer interaction. However, the relatively limited number of studies indicates that these technologies remain at an early stage of adoption in health care, likely due to higher development costs, hardware requirements, and implementation complexity.

Although AI was increasingly incorporated into HCAs, most of these implementations appear to remain at a relatively immature stage, which is consistent with other studies demonstrating the difficulties of translating AI implementations into clinical workflows [225-227].

SUS clearly dominated the assessment of HCAs, despite having been developed for generic interactive systems. SUS is widely used for usability evaluation of different types of digital health applications [215,228-230] because it is an easy and quick tool with a scoring system that has been translated into multiple languages [228]. However, it was not originally developed to assess complex multimodal interaction mechanisms and does not capture the holistic user experience because it focuses only on usability aspects such as ease of use, satisfaction, and learnability.

Although several alternative instruments were identified, CUQ was the only instrument specifically appropriate for assessing HCAs, even though it does not fully capture the multimodal and AI-driven characteristics of modern HCAs. The absence of recently developed instruments [46], such as the Conversational Agents Scale [231], the Conversational Agent’s Usage Scale [232], the User Experience Evaluation of Conversational Agents [233], or the Conversational Agent Scale for User Experience [234], further highlights the need for greater methodological standardization in future evaluations. Moreover, a relevant number of studies were excluded from this review because usability and user experience assessments were based on questionnaires consisting of Likert scale questions developed by the researchers. However, these questionnaires were not validated instruments because they had not undergone psychometric analysis, which is frequently a lengthy process that can be particularly challenging for researchers who lack experience in developing and validating subjective instruments.

Usability and user experience assessments were conducted in both controlled (ie, research or academic settings, outpatient clinical settings, hospital clinical settings, and institutional residences for older adults) and uncontrolled (eg, participants’ homes) environments. Controlled experiments generally focus on intensive sessions to capture immediate usability or user experience feedback. These experiments enable consistent data collection, but they may constrain the observation of more natural interaction patterns. By contrast, longitudinal studies conducted in uncontrolled settings allow participants to engage with HCAs more organically, integrating their use into daily routines. Therefore, uncontrolled environments provide authentic user experiences and valuable insights, as well as functional challenges, including connectivity issues, reflecting authentic usage scenarios. Moreover, many longitudinal studies strategically combined uncontrolled environments with data collection in controlled environments.

The participants comprised both patients and healthy adults or older adults. However, the reviewed studies often did not adequately justify their choice of sample sizes. Small samples can undermine the reliability of results by increasing the risk of sampling bias and reducing statistical power. Consequently, the findings may not capture the full variability of the population, thereby restricting their generalizability.

The quality assessment of the components related to the application of subjective measurement instruments highlighted methodological limitations, including insufficient evaluation of prolonged real-world use and inadequate justification of sample sizes, the 2 CAUSS items with lower percentages. These shortcomings reduce confidence in the ecological validity and generalizability of the reported usability findings.

Looking at the assessment results, the usability and user experience of HCAs tend to be above average; however, the results indicate differences among the various HCA categories. The comparison of usability scores across the 3 HCA modalities revealed statistically significant differences, with pairwise comparisons indicating that text-based HCAs achieved significantly higher usability scores than both voice-based and embodied HCAs, whereas no significant differences were observed between voice-based and embodied HCAs. The estimated eta-squared (η²H=0.11) indicates a moderate effect size, suggesting a moderate association between interaction modality and users’ perceptions of usability. No significant differences were found in the number of participants across the 3 HCA categories, indicating that the studies included broadly comparable sample sizes. Consequently, systematic differences in study size are unlikely to fully explain the observed differences in usability.

Several factors may contribute to the higher usability observed for text-based HCAs. Although the reviewed studies did not directly investigate the underlying mechanisms, one possible explanation is users’ greater familiarity with text-based communication, particularly through messaging applications. Moreover, text interfaces provide a persistent conversation history that enables users to review previous exchanges, potentially increasing their sense of control and reducing memory demands during interaction. In addition, text-based interactions allow users to engage with HCAs at their own pace, giving them more time to read, reflect on, and formulate responses. By contrast, voice-based HCAs may be affected by speech recognition inaccuracies, environmental noise, and privacy concerns, whereas embodied HCAs introduce additional visual and social interaction elements that may increase interaction complexity without necessarily improving perceived usability.

The results also suggest that older adults tend to perceive HCAs as less usable than adults. This may be associated with older adults generally having lower digital literacy, as well as age-related accessibility needs. For example, age-related sensory, cognitive, and motor changes, together with the higher prevalence of multimorbidity among older adults, may affect their ability to interact effectively with HCAs. Nevertheless, the available evidence from the included studies does not allow this explanation to be confirmed.

Interestingly, the difference between adults and older adults was statistically significant only for text-based HCAs, despite this modality achieving the highest usability scores overall. A potential explanation is that text-based HCAs are currently more widespread than voice-based or embodied HCAs, which may lead to greater differences in familiarity and prior experience between age groups.

Although no statistically significant age-related differences were observed for voice-based or embodied HCAs, the estimated effect sizes were of comparable magnitude across modalities. Therefore, the absence of statistical significance may reflect the smaller number of available studies and limited statistical power rather than provide evidence that age has no influence on perceived usability in these interaction modalities.

These results should be interpreted with caution. The included studies exhibited substantial heterogeneity in terms of application domains (patient care, health education and prevention, health data collection, and support for daily activities), study designs (eg, cross-sectional or longitudinal), evaluation contexts (controlled versus real-world settings), target populations, underlying technologies (eg, AI, large language models, or virtual reality), task complexity, and duration of HCA use (single-session vs prolonged use). Consequently, the observed differences in SUS scores may have been influenced by factors beyond the interaction modality itself. Furthermore, the SUS assesses perceived usability rather than objective performance or long-term acceptance, and therefore high SUS scores do not necessarily translate into sustained engagement or continued use. Future studies should investigate which specific design characteristics contribute most to usability across different HCA modalities while accounting for user characteristics and contextual factors.

Despite the use of extensive search strategies to locate relevant studies, some articles may have been overlooked due to variations in terminology. Furthermore, the exclusion of non-English publications and gray literature may have limited the overall representativeness of the findings and introduced language and publication bias, as studies published in other languages or unpublished studies with null or negative findings may have been underrepresented.

The wide variety of assessment instruments used meant that it was generally not possible to identify enough studies to enable a comparative analysis of usability or user experience outcomes by instrument across different HCA categories. The only exception was SUS; however, as it was developed in the 1990s, it may not adequately capture relevant aspects of the usability of modern HCAs.

Finally, this review focused solely on the usability and user experience of HCAs and did not address their efficacy or effectiveness.

In conclusion, this review included 127 studies applying validated subjective instruments to assess the usability and user experience of 3 HCA categories: text-based, voice-based, and embodied HCAs. In terms of health care applications, HCAs aimed to support patient care, health education and prevention, health data collection, and daily activities of older adults.

The usability and user experience of HCAs were assessed in cross-sectional and longitudinal studies using the following instruments: SUS, UEQ, UEQ-S, CUQ, AES, MAUQ, UMUX-Lite, USE, CSUQ, and PSSUQ.

This review highlights that, except for CUQ, the identified subjective assessment instruments were not specifically developed to evaluate the usability and user experience of HCAs. Therefore, the findings suggest that such evaluations may be limited by the inadequacy of these instruments.

Considering the comparative analysis of SUS scores, text-based HCAs tend to show higher usability scores than voice-based and embodied HCAs, and older adults tend to perceive HCAs as less usable than adults. However, these findings represent associations observed across the included studies rather than direct evidence that one HCA category is inherently more usable than another, because the studies differ substantially in their health care applications, design, evaluation contexts, participant populations, and implementation and use characteristics. Moreover, a relevant number of studies used small sample sizes, which reduce statistical power, may fail to capture population variability, limit the generalizability of the results, and increase the risk of misleading conclusions. Most HCAs were also assessed in controlled environments at a single point in time, without considering the impact of continuous and long-term use in real-world settings.

Considering the substantial differences in how evaluations were designed and conducted across studies, especially regarding data collection methods, participant selection, testing conditions, and reporting of results, there is a need for standardized protocols and established guidelines to measure the usability and user experience of HCAs. Such standardization would improve methodological consistency and enhance the comparability of findings across studies, enabling researchers and practitioners to benchmark usability and user experience.