The Health-ITUES consists of 20 items and can be customized according to the characteristics of the device and the intended use. Originally, it was developed based on the Technology Acceptance Model (TAM) to evaluate the usability of a web-based scheduling system for nursing staff [23]. Subsequently, Schnall et al [27] further validated the scale using a self-management application for individuals living with HIV, focusing on the evaluation of self-management mHealth applications.

In this study, we customized the Health-ITUES for young women experiencing anxiety related to STIs, based on the validated versions used among HIV-positive individuals in the United States and women with menstrual discomfort in South Korea. Prior to initiating the translation and adaptation process, we obtained permission from the original author to develop a Japanese version of the Health-ITUES.

The translation process adhered to established guidelines for scale translation [30] and followed the sequence as described in the following sections.

We conducted an online survey using a chatbot under development by the authors to evaluate the reliability and validity of the Japanese version of the Health-ITUES. The chatbot was designed to alleviate anxiety related to STIs among young women and to guide them toward appropriate consultation services and testing resources. Internal consistency reliability was assessed using the Cronbach α coefficient. Construct validity was examined through confirmatory factor analysis (CFA), and criterion-related validity was evaluated by examining correlations with user satisfaction and intention to use.

Participants were women aged 18 to 29 years registered with a Japanese research agency (Asmarq Inc). According to national infectious disease surveillance data [33], reported syphilis cases—one of the major STIs—among women in their 20s have increased substantially since 2020, making this demographic particularly relevant for the chatbot’s intended use. Eligibility criteria included the ability to communicate in Japanese, access the survey URL using personal computers or smartphones, and have had at least one prior sexual experience. Individuals who were health care professionals, medical students, or already engaged with STI-related medical services or consultation centers were excluded from the study.

To ensure adequate statistical power in scale validation studies, a commonly used rule of thumb is to recruit approximately 10 participants per item [32,34]. Given that the Health-ITUES consists of 20 items, we estimated the minimum required sample size to be 200 participants. Assuming a 20% dropout rate, the target sample size was set at 250 participants.

Participants were recruited through a research agency. The agency randomly selected 57,185 women within the target age range from its registered panel and sent email invitations to participate in the study. Only those who provided informed consent via an embedded survey link were directed to the main questionnaire. The survey was terminated once the number of responses exceeded the target sample size. Data were collected between June 18, 2025, and June 20, 2025. A total of 301 valid responses were obtained.

Participants were informed via a web-based survey interface that the study involved interacting with a chatbot based on a hypothetical scenario (ie, their partner suddenly informed them of symptoms suggestive of an STI). Instructions on how to use the chatbot were also provided. Initially, participants completed prechatbot questions on age, region of residence, occupation, educational background, relationship status, frequency of conversational AI use, and digital health literacy [35]. They then accessed the chatbot interface through a URL embedded in the survey and interacted with it according to the scenario. After the interaction, participants returned to the survey and completed postchatbot questions assessing satisfaction, intention to use, and the Japanese version of the Health-ITUES.

The survey included the following items:

We calculated the mean scores and SDs for the overall Japanese version of the Health-ITUES and its subscales (impact, perceived usefulness, ease of use, and user control). Ceiling and floor effects (acceptable if <15% at maximum and minimum scores) were examined. We assessed reliability through internal consistency, using the Cronbach α coefficient and composite reliability (CR). Convergent validity was evaluated based on the average variance extracted (AVE). To assess construct validity, a CFA was conducted according to the original 4-factor model. Model fit was evaluated using fit indices, including the Goodness-of-Fit Index (GFI), Comparative Fit Index (CFI), Tucker-Lewis Index (TLI), root mean square error of approximation (RMSEA), and standardized root mean square residual (SRMR). Criterion-related validity was assessed by examining the correlation coefficients between Health-ITUES scores and conceptually related variables, namely user satisfaction and intention to use the chatbot. All statistical tests were conducted at a 5% significance level using IBM SPSS Statistics version 29.0 and AMOS version 31.0.

This study was approved by the Research Ethics Committee for human participants at Teikyo University (approval numbers: Tei-Rin 24‐114, dated January 30, 2024, and Tei-Rin 25‐006, dated May 8, 2025). Participants provided informed consent via an embedded survey link. Participants received a token of appreciation, equivalent to 200 Japanese yen (1 Japanese yen=US $0.006 as of May 8, 2025) in points, for their participation in the study.

We conducted a pilot test with 6 female university students aged 18 to 22 years, recruited through convenience sampling. All participants were native Japanese speakers and nonmedical majors. They completed the Japanese version of the Health-ITUES and were subsequently interviewed regarding the clarity of language, ease of response, and confidence in their responses. One participant commented, “There were no particular difficulties in responding, but because the chatbot was accessed via a URL rather than through a login, it might be better to revise that wording.” Based on this feedback, we revised item 17 to clarify that the chatbot was accessed via a URL in this study.

Table 2 presents the demographic characteristics of the 301 participants. Their mean age was 26.3 (SD 2.1) years. Although more than half of the participants were company employees, the sample included a diverse range of occupations, such as students, homemakers, and part-time workers. The most common frequency of conversational AI use was “occasionally” (n=109, 36.2%), followed by “never,” “rarely,” and “almost daily,” each accounting for approximately 20% of responses. When “occasionally” and “almost daily” users are combined, approximately 60% (n=177) of participants reported using conversational AI to some extent, which is consistent with national data indicating that around 50% of young people in Japan use such technologies [36].

The average score on the 15-item Digital Health Literacy Scale (DHLS) was 2.8 (SD 0.4). This result aligns closely with the findings from a study of Japanese university students, which reported an average score of 2.9 (SD 0.4) on the 21-item DHLS [37]. These findings indicate that the study sample is broadly representative of young Japanese individuals in terms of digital health literacy.

The Health-ITUES consists of 20 items across 4 subscales (impact: 3 items, perceived usefulness: 9 items, ease of use: 5 items, and user control: 3 items), each rated on a 5-point Likert scale (1=“strongly disagree” to 5=“strongly agree”). Satisfaction was assessed on a 5-point scale (1=“not satisfied at all” to 5=“very satisfied”), along with intention to use (1=“not at all willing to use” to 5=“very willing to use”). The overall mean score for the Japanese version of Health-ITUES was 3.4 (SD 0.9), whereas user satisfaction was 3.1 (SD 1.2) and intention to use was 3.2 (SD 1.2; Table 3). Among the subscales, user control showed slightly lower mean scores, although no substantial differences were observed. No ceiling or floor effects were detected.

The internal consistency and convergent validity were evaluated, with Cronbach α of more than 0.8, CR exceeding 0.70, and AVE exceeding 0.50. As shown in Table 3, Cronbach α coefficients for the subscales ranged from 0.85 to 0.98, indicating high reliability beyond the commonly accepted cutoff of 0.80. The impact and perceived usefulness subscales had particularly high α values (>0.95). The overall Cronbach α for all 20 items was 0.976, which may be attributed to the large number of items in the usefulness subscale and high intersubscale correlations. Descriptive statistics of the 20 items (mean, SD, skewness, kurtosis, and item-total correlations) are provided in Multimedia Appendix 1.

As the conceptual validity of the original Health-ITUES had already been established, we performed a CFA using maximum likelihood estimation based on the original 4-factor structure. The analysis provided standardized factor loadings to assess the strength of the relationship between items and factors, along with model fit indices.

The path diagram is shown in Figure 1, and Table 4 presents the model fit indices. The CFA results indicated an adequate model fit (CFI=0.94, TLI=0.93, and SRMR=0.032). However, the GFI was relatively low (0.8), and the RMSEA was 0.100, representing the upper limit of the acceptable range [26].

Intersubscale correlations ranged from 0.69 to 0.93, indicating strong relationships. In particular, perceived usefulness showed a higher correlation with impact, which is consistent with the findings of previous research [25,26,29].

We examined criterion-related validity by calculating the correlation coefficients between Health-ITUES scores and conceptually related variables, namely user satisfaction and intention to use. The correlations were 0.779 and 0.797, respectively, indicating a strong positive association (P<.001).

The emergence of generative AI has expanded the potential of mHealth. Chatbots are increasingly recognized as promising tools for supporting STI prevention and sexual health education among young people, and their role is expected to grow [11,38-41]. At the same time, usability—including user satisfaction and ease of use—remains critical for promoting continued engagement and facilitating behavior change [39,40].

The instrument selected for this study—the Health-ITUES—was developed as a user-centered evaluation scale based on the Health-ITUEM, which itself was derived from the widely used TAM and specifically adapted for health care technologies [23,28].

This study developed a Japanese version of the Health-ITUES and examined its reliability and validity using a chatbot currently under development as a consultation tool for young women with concerns about STIs. Because the original scale was designed to evaluate an app for adults living with HIV, we adapted the item wording to align with the chatbot’s purpose and functions rather than conducting a direct translation.

In this study, we confirmed that the 4-factor structure of the Health-ITUES—impact, usefulness, ease of use, and user control—was largely retained in the context of a Japanese STI-related chatbot.

The translation procedures followed established guidelines, including forward translation by 2 experts in medicine and health communication, review by a multidisciplinary team of physicians and health communication experts, and a back-translation performed by a medical translation specialist. Cognitive debriefing and a final review by one of the authors, who validated the original Health-ITUES, ensured the cultural appropriateness and conceptual consistency of the translated version.

The customized Health-ITUES demonstrated adequate reliability and convergent validity for evaluating the usability of the STI-related scenario chatbot, and the main model-fit indices were satisfactory. In the CFA, although the RMSEA was slightly higher, the overall model fit was still considered acceptable because the other indices indicated a generally adequate level of fit [26]. Strong positive correlations between Health-ITUES scores and both user satisfaction and intention to use further supported criterion validity. These findings indicate that the Health-ITUES supports item-level customization while remaining suitable for multilingual translation and cultural adaptation.

The psychometric evaluation indicated very high internal consistency for the impact and usefulness subscales (Cronbach α>0.95). Although such values reflect strong reliability, they also suggest that some items may capture overlapping aspects of the same construct [42]. This interpretation is supported by the relatively strong interitem correlations observed within these subscales (r>0.85). Although these patterns suggest potential item redundancy and warrant future examination of item reduction or refinement, it is important to note that the Health-ITUES was intentionally designed to allow item-level customization while preserving conceptual comparability across contexts. The higher correlation between impact and perceived usefulness was also observed in previous studies [25,26,29]. Consistent with this, we retained all 20 items in the Japanese version without modifications or deletions.

From a theoretical perspective, this study reinforces the importance of usability as a multidimensional construct in digital health and demonstrates that the 4-factor structure of the Health-ITUES remains stable even when adapted to conversational AI contexts. Methodologically, this study highlights the feasibility of combining item-level customization with rigorous cross-cultural translation procedures while preserving conceptual equivalence. This approach provides a model for adapting usability instruments to rapidly evolving digital health technologies. To date, no other Japanese usability scale has demonstrated conceptual comparability while allowing flexible item-level customization tailored to specific use cases. The Japanese version of the Health-ITUES is expected to serve as a valuable tool for assessing the usability of mHealth and conversational AI applications as their use expands.

The overall mean Health-ITUES score in this study was 3.4 (SD 0.9), which was lower than that reported in previous studies [24,25]. Several contextual factors may explain this difference. Our evaluation used a chatbot that is still in the development phase, which may have influenced perceptions of quality and usability. In contrast, earlier studies assessed mature, widely used applications developed for long-term engagement by specific patient populations, such as medication adherence support for individuals with HIV [27] or symptom management for dysmenorrhea [25], which may have already achieved high user acceptance. Moreover, this study involved a single-use interaction based on a hypothetical scenario rather than a real-world health context, which may have contributed to the lower scores.

At the same time, these findings highlight a practical advantage of the Health-ITUES: it can be used not only to evaluate relatively mature digital health tools but also to assess prototypes and guide iterative improvement during early-stage development. Future evaluations of the same chatbot using the Health-ITUES under conditions of continued use may help clarify how perceived usability evolves as interactions between users and conversational AI systems accumulate over time. Based on user feedback, we are currently developing an improved version of the chatbot by enhancing its frequently asked question responses and refining error messages. Incorporating the Health-ITUES from the early stages of development may support user-centered improvements and provide an objective basis for monitoring usability changes over time.

From a practical standpoint, the availability of a validated Japanese version of the Health-ITUES enables more systematic and standardized usability evaluations of mHealth and conversational AI tools in Japan.

A key strength of this study was its robust sample size and the diversity of participants in terms of region and occupation. The sample size of 301 participants in our study exceeded the commonly used 10 times per item rule [32,34]. The mean score on the 15-item DHLS (2.8, SD 0.4) aligned with that of previous Japanese studies [37], and the distribution of conversational AI use showed no marked skew, supporting the appropriateness of the sample for usability assessment in this context.

This study had some limitations. First, participants were not actual patients or individuals currently experiencing health concerns; instead, we conducted the usability evaluation under a hypothetical scenario. Therefore, the results may not fully reflect real-world usage, and future research should assess its usability in clinical or personal health contexts. Second, the study population was limited to young women. Although we selected this demographic to align with the intended application of the mHealth chatbot, it is important to examine whether the Health-ITUES remains a valid usability assessment tool across other user groups in Japan, such as adult men, older adults, and individuals with chronic conditions, to enhance its generalizability.

This study presented the first Japanese version of the Health-ITUES and provides initial evidence of its reliability and validity in the context of an STI-related chatbot. To our knowledge, no other Japanese usability scale for mHealth combines validated psychometric properties with the flexibility to be customized according to specific use cases. By preserving conceptual equivalence while enabling standardized, user-centered usability evaluation, the Japanese version of the Health-ITUES addresses a critical gap in digital health assessment in Japan. As conversational AI and mHealth applications continue to expand, the Japanese version of the Health-ITUES may serve as a valuable tool for supporting the development, evaluation, and implementation of effective and user-centered digital health interventions.