Following authorization from the original scale’s corresponding author, the translation of the BUS-11 into Chinese was conducted using the Brislin translation model as improved by Jones et al [18]. Consistent with established guidance for cross-cultural instrument adaptation (eg, forward-back translation with expert reconciliation), we documented all translation decisions to ensure semantic and conceptual equivalence. The specific steps are as follows: (1) forward translation: the researcher and a bilingual professional proficient in both Chinese and English independently translated the original BUS-11 into Chinese. Discrepancies and alternative phrasings were recorded in a reconciliation log and resolved through consensus, resulting in an initial Chinese draft; (2) back translation: another bilingual translator, whose native language was Chinese, translated the initial Chinese draft back into English, producing a back-translated English version; (3) consistency check: a monolingual English-speaking expert familiar with the BUS-11 reviewed and compared the original scale with the back-translated version. Any residual inconsistencies (eg, colloquialisms or culture-specific expressions) were discussed in joint meetings with all translators until consensus was achieved, ensuring conceptual and semantic consistency; (4) final translation: a final Chinese translation was prepared collaboratively by all translators, incorporating adjustments from the consistency review, thus establishing semantic equivalence with the original BUS-11. Minor wording refinements were made for clarity and idiomatic naturalness (no item content was substantively altered). At the end of the cultural adaptation process, the final Chinese version of the BUS-11 and the original English items are presented side by side in Table 1 to facilitate direct comparison of item wording and to enhance the transparency of the adaptation procedure.

In June 2024, we convened an expert committee (n=6) to evaluate the cultural appropriateness of the Chinese translation. The expert committee comprised 6 specialists drawn from the broader panel reported in Table 2, all of whom had doctoral-level training or senior professional experience in HCI, usability or UX evaluation, health informatics, or scale development. Eligibility criteria included: (1) formal training in a relevant discipline, (2) demonstrated experience in chatbot-related research or usability evaluation, and (3) prior involvement in scale development, instrument adaptation, or related methodological work. Each expert independently compared the Chinese items with the original BUS-11 and assessed semantic and conceptual equivalence, idiomatic appropriateness, and readability. Disagreements were resolved through iterative discussion until consensus was reached. Revisions focused on wording-level refinements to improve clarity and align the tone with standard Chinese conversational norms, without altering the intended coverage of the construct.

To ensure participants evaluated chatbots after a realistic, standardized interaction, we designed representative use-case tasks for 10 commonly used chatbot systems in China (refer to Table 3 for details). For example, participants were instructed to interact with Taobao’s intelligent customer service, simulating a typical query related to membership benefits: “Your friend recommends you purchase Taobao’s 88vip. Please use Taobao’s chatbot ‘Smart Customer Service Xiao Mi’ to help you understand the specific benefits of 88vip. Log in as instructed and converse with this chatbot.” Refer to Multimedia Appendix 1 for details. Immediately after completing the assigned task, participants filled out the Chinese BUS-11 about that chatbot. The same task-rating flow was later applied in the main study.

In July 2024, a pilot study involving 15 participants was conducted at a Beijing-based company, using convenience sampling. Participants completed the Chinese BUS-11 after performing designated tasks with selected chatbots. Completion times were recorded, and qualitative feedback regarding item clarity, ambiguity, and comprehension difficulties was collected through structured interviews. The pilot’s primary aims were to assess task feasibility, confirm average completion time, and identify any items requiring wording refinements. Pilot data were used only for refinement and were not included in the main validation dataset.

Based on the observed average completion time (~5 minutes 20 seconds), we set a priori a data-quality rule for the main study to exclude online questionnaires completed in less than 5 minutes, in addition to removing any incomplete submissions.

Participants were recruited via social media platforms (eg, Red Note [Xiaohongshu Information Technology] and WeChat [Tencent Holdings Ltd]) according to predefined inclusion criteria: native Chinese speakers or individuals proficient in Chinese and aged 18 years or older. Recruitment followed an online convenience sampling approach; participation was voluntary. Each participant was randomly assigned to interact with 3 distinct chatbots (selected from the 10 systems), performing 1 predefined task per chatbot and completed the BUS-11 immediately after each interaction. Thus, each participant contributed 3 independent chatbot evaluations, yielding a repeated-measures data structure (evaluations nested within participants). For psychometric analyses (eg, reliability and exploratory factor analysis [EFA]), evaluations were analyzed at the evaluation level (N=438); design implications of repeated measures are discussed in the “Limitations” section.

Based on factor analysis requirements (5‐10 times the number of items, accounting for a 20% invalid rate), a minimum sample size of 69 participants was established. We targeted ≥70 participants and exceeded this target, ultimately recruiting 214 participants who provided 438 evaluations, thereby improving the robustness of reliability and factor-analytic estimates.

We analyzed 438 valid chatbot evaluations submitted by 214 participants (69 males and 145 females), with each participant randomly assigned to evaluate 3 of 10 popular Chinese chatbot systems using the Chinese BUS-11. The sample skewed young, with 67.3% (144/214) aged 18‐24 years and 22.4% (48/214) aged 25‐30 years, and was highly educated, with 93% (199/214) holding a bachelor’s degree or above, and represented multiple industries, most prominently health care (52.8%, 113/214) and education or training (11.7%, 25/214). Most respondents had prior exposure to chatbots (79%, 169/214) and at least moderate familiarity (43%, 92/214) reported being familiar or very familiar, while 27.5% (59/214) reported being unfamiliar or very unfamiliar. These characteristics suggest a relatively tech-savvy cohort, which should be considered when interpreting generalizability. Detailed distributions are provided in Table 4.

Two primary instruments were used:

Data were collected electronically via Tencent Questionnaire. Participants completed tasks on their own devices and then accessed the BUS-11 through a survey link. Strict criteria ensured data quality, and questionnaires that were not fully completed were directly discarded. Based on pilot timing, online questionnaires completed in less than 5 minutes were excluded to reduce insufficient-effort responses. In the pilot experiment, the research team measured the time taken to complete the questionnaire, with an average duration of 5 minutes and 20 seconds. The finalized main study dataset comprised 438 valid chatbot evaluations.

Data were exported to Microsoft Excel, verified, and analyzed using IBM SPSS Statistics (version 26.0). Reliability was evaluated using Cronbach α coefficients for the overall scale and for each of its multi-item dimensions. Structural validity was assessed via EFA after sampling-adequacy checks (Kaiser-Meyer-Olkin [KMO] index and Bartlett test of sphericity). We used a principal components extraction with the Kaiser criterion (eigenvalues >1.0) and scree plot inspection to determine the number of factors and applied varimax rotation to aid interpretability. Standard criteria guided item retention: primary factor loadings of ≥0.40, absence of substantial cross-loadings (≥0.30), and conceptual interpretability of the resulting factor solution. Salient loadings were defined as those with values of ≥0.40. In line with the instrument’s structure, the EFA focused on the 9 items related to multi-item dimensions; the 2 single-item dimensions (“Privacy and Security” and “Response Time”) were not included in the EFA but were retained as single-item indicators in descriptive analyses. P<.05 was considered statistically significant for inferential tests (eg, Bartlett test of sphericity). We note that principal component analysis (PCA) is commonly used in early-stage exploratory work for data reduction and structure discovery, and future work may complement this with common-factor methods (eg, principal axis factoring) and confirmatory factor analysis (CFA) once larger samples are available.

Before factor extraction, data suitability was confirmed (KMO=0.743; Bartlett χ²₃₆=628.185; P<.001) because 2 BUS-11 dimensions are single-item constructs (“Privacy and Security” and “Response Time”) and they were not included in the EFA. The analysis, therefore, used the remaining 9 items (refer to Figure 1 for the scree plot).

In this study, PCA and varimax orthogonal rotation were used to extract common factors. Nine items (items 10 and 11 were excluded from the EFA due to having only 1 item each) were analyzed and combined with the scree plot (Figure 1), and 3 common factors with eigenvalues greater than 1.0 were ultimately extracted, accounting for a cumulative variance contribution rate of 56.107%. The results of the EFA for this experiment are shown in Table 5.

EFA with varimax rotation revealed a clear 3-factor structure for the Chinese version of BUS-11 (Table 5). The 3 factors explained 15.2%, 19.6%, and 21.3% of the total variance, respectively, accounting for a cumulative 56.1% of the variance. Factor 1 (Accessibility) consisted of 2 items reflecting the ease of locating and using chatbot functions, with strong loadings (0.840 and 0.760). Factor 2 (Interaction Process Quality) consisted of 3 items that described conversational smoothness, coherence, and comprehensibility, with loadings ranging from 0.716 to 0.759. Factor 3 (Information Quality) consisted of 4 items related to understanding user intent, the appropriateness of information, the absence of irrelevant content, and the reliability of responses, with loadings ranging from 0.627 to 0.739.

Only factors extracted by the eigenvalue >1 criterion are shown. Item domain labels reflect the original BUS-11 conceptual domains and are provided for reference.

The overall internal consistency of the Chinese BUS-11 was excellent (Cronbach α=0.92). By factor, reliability was acceptable to excellent, as indicated by Accessibility (Cronbach α=0.79), Interaction Process Quality (Cronbach α=0.91), and Information Quality (Cronbach α=0.93). Reliability was not computed for the single-item dimensions (“Privacy and Security” and “Response Time”). Item-level diagnostics (Table 6) showed corrected item-total correlations ranging from 0.453 to 0.819, indicating that all items were positively associated with the overall scale score. Cronbach α (if an item was deleted) ranged from 0.91 to 0.93, suggesting that removing any single item would not meaningfully improve internal consistency.

We conducted a comprehensive psychometric evaluation of the Chinese BUS-11, including assessments of content validity, construct (factorial) validity, internal consistency reliability, and user feedback on item clarity and practical usability. Overall, the findings provide strong evidence that the localized scale retains the scientific integrity of the original instrument. Here, we discuss each set of findings in turn, relating them to established benchmarks and prior research, and interpret their implications for the scale’s use in HCI and human factors studies.

By consulting a panel of 8 experts to review the relevance of each item, we established that the content validity of the Chinese BUS-11 is high. The I-CVI for every item was very strong, with 4 items (Items 3, 4, 6, and 11) achieving an I-CVI of 1 (ie, unanimously deemed highly relevant by all experts). The remaining items also scored well above conventional acceptability thresholds [25]. Given that an I-CVI≥0.78 is often considered the minimum when 6 or more experts are involved, our results indicate that each item is considered clearly representative of the intended construct by domain specialists. The S-CVI was 0.920 (both by the averaging method and the universal agreement method), exceeding the common cutoff of 0.90 for excellent content coverage of a construct. These indices confirm that the BUS-11 (Chinese version) has comprehensive content coverage and the items collectively cover the important facets of chatbot usability without obvious omissions. This outcome was facilitated by the diverse expertise of our review panel, which included professionals in HCI, usability engineering, and systems engineering. Their varied perspectives helped ensure that the items were not only translated correctly but also conceptually appropriate for Chinese users. In practical terms, the high content validity suggests that Chinese BUS-11 can confidently be used to gauge chatbot usability aspects as intended—such as efficiency, clarity of answers, and user comfort—without missing key elements. The rigorous content validation process, in line with recommended scale development procedures, provides a solid foundation for the subsequent construct validation. It also offers reassurance to practitioners that the instrument has face validity in the local context (the items make sense to experts and presumably to end-users as well). We believe this strong content validity will help drive adoption of the scale in both research and industry evaluations, as stakeholders can trust that the instrument measures what it is supposed to measure (ie, salient user experience factors for chatbots).

To examine the underlying factor structure of the Chinese BUS-11, we carried out an EFA on the survey responses. Preliminary checks confirmed that our data were suitable for factor analysis: the KMO measure of sampling adequacy was 0.743, which is above the usual threshold of 0.60 and can be considered “middling” to “good” [26], and the Bartlett test of sphericity was highly significant, indicating sufficient interitem correlations for extracting latent factors. Using principal component extraction with varimax rotation, we identified a 3-factor solution that best fit the data. These 3 extracted common factors had eigenvalues greater than 1 and together accounted for about 56.1% of the total variance in users’ responses. A cumulative variance above approximately 50% is acceptable in behavioral research, given the complexity of HCI measures, so 56% indicates that the scale captures a substantial portion of the usability perception variance in our sample. Each factor showed a clear thematic grouping of items, reflecting distinct dimensions of the chatbot user experience. Notably, 2 specific usability aspects (ie, “Privacy and Security” and “Response Time”) were each originally measured by a single item in the BUS-11. As single-item facets, they could not load onto multi-item factors in the EFA and were thus analyzed separately (we examined their scores independently rather than including them in the factor structure). The remaining nine items clustered into three coherent factors: (1) Accessibility (eg, ease of accessing and using the chatbot, such as simple interface and low effort to start the interaction), (2) Interaction Quality (covering the smoothness, coherence, and understandability of the conversational exchange), and (3) Information Quality (assessing the relevance, clarity, and correctness of the chatbot’s responses, without unnecessary redundancy). This structure is broadly consistent with the original conceptual framework of the BUS scale, which encompassed multiple dimensions of chatbot usability [10], but our findings suggest a somewhat more consolidated factor model in the Chinese context.

In the original development of the BUS, a 5-factor model (BUS-15) was proposed, which included separate dimensions for privacy and response speed alongside factors analogous to those we found [10]. Our EFA results imply that Chinese users may not distinguish as many separate categories, instead perceiving a more integrated set of usability factors. Attributes related to the conversational process and functional outcome appear to intertwine; users who find the interaction process smooth also tend to perceive the information provided as high quality, suggesting an underlying general perception of “interaction effectiveness.” Interestingly, a recent re-examination of BUS-11 in a large multichatbot dataset also identified a simpler factor structure than initially theorized [27]. Our findings align with that perspective, reinforcing the idea that a parsimonious model (fewer and broader factors) can sufficiently capture chatbot usability evaluations. It is worth noting that we chose to retain the “Privacy and Security” and “Response Time” items as standalone indicators in the instrument, due to their importance in usability (users do care about privacy and speed) and their presence in the original scale. However, from a psychometric standpoint, single-item factors are suboptimal because they do not allow for internal consistency reliability estimation and cannot capture the breadth of a construct. Best practices in scale development typically recommend having at least 2‐3 items per factor to achieve adequate reliability and construct representation [28]. Therefore, one implication is that future refinement of the Chinese BUS might involve expanding these facets—adding a couple of items to better gauge the privacy/security aspect of chatbot use (eg, covering data protection, user consent, and so on) and the responsiveness aspect (eg, not just speed but also perceived responsiveness or promptness). For now, including those single items ensures that our localized scale does not lose any content relative to BUS-11, and they provide useful standalone measures (eg, designers might specifically want to see ratings of privacy transparency). In summary, the EFA supports the structural validity of the Chinese BUS-11 by confirming that its items coherently measure multiple distinct dimensions of usability. The identified 3-factor structure captures the major themes of chatbot usability experience in our sample. This structure provides an empirical basis for scoring or interpreting the scale (eg, computing subscale scores for Accessibility, Interaction Quality, and Information Quality) and sets the stage for future confirmatory testing to verify whether this structure generalizes to other samples or holds under more stringent statistical criteria.

The observed variability in interpreting the privacy/security item suggests that privacy-related usability judgments may be more sensitive to users’ privacy literacy and to how risk information is surfaced in routine interactions, consistent with prior privacy research indicating limited user engagement with privacy policies and disclosures [29,30]. From a measurement perspective, this supports retaining privacy/security as a relevant usability facet while motivating future refinement (eg, adding additional items or more concrete wording) and the use of cognitive interviewing to improve interpretability across user subgroups [31]. In addition, the brief completion time (≈3‐4 minutes) suggests that the Chinese BUS-11 imposes low respondent burden, comparable to widely used usability instruments and suitable for rapid assessment in applied settings [32,33].