This study was approved by the Monash University Human Research Ethics Committee (ID 26129), which encompassed research conducted in both Australia and Canada. Chatbot users were offered the opportunity to complete web-based surveys throughout their conversation with the JEM chatbot; however, the surveys were framed as completely optional, and users could ignore the survey and continue using the chatbot without consequence. When users clicked on a survey weblink, they were presented with detailed study information. Users were informed of their right to withdraw from the survey at any time without any negative impacts. After the detailed information, users were asked to consent by clicking on a button that they had read the information and consented to proceeding to the survey. For users aged 13 to 15 years, ethics-approved procedures required them to confirm that they had obtained parental or guardian consent via an online acknowledgment. Given the fully anonymous and low-risk nature of the research, this approach was deemed appropriate and proportionate by the approving committee. The lower age threshold of 13 years was selected in line with common minimum ages for digital platform access and social media use and to capture early adolescent users who may engage with online eating disorder–related content. Users aged 16 and 17, like those aged 18 and over, were considered to be sufficiently mature to consent for themselves without parent or guardian consent required according to our ethics approval. Users completed the surveys anonymously—they were not required to provide their name or any potentially identifying information at any point. No compensation was offered for completing surveys or using the chatbot.

This study used a naturalistic mixed methods design comprising 2 complementary data structures: (1) cross-sectional user evaluation surveys and (2) within-session pre-post microintervention assessments embedded within chatbot interactions. A minimum of 34 completed responses per microintervention was used as a pragmatic threshold intended to support estimation of moderate effect sizes [37], consistent with prior work [31,34], and the exploratory nature of this real-world design [38].

Participants were individuals aged 13 years and older residing in Australia or Canada who engaged with the JEM chatbot from January 2023 to June 2023 in Australia or September 2024 to March 2025 in Canada. Geographic location was based on self-reported residence and deployment platform context within an anonymous web-based design. These time periods covered the first 6 months of JEM deployment in each country for optimal comparability, particularly in terms of the novelty of the chatbot service to the users in the respective countries [39]. In this 6-month period, system-level usage included 6862 total chatbot sessions and 39,184 interactions in the Australian deployment and 1332 sessions and 5370 interactions for the Canadian deployment according to Google Dialogflow analytics definitions [40], which are distinct from individual-level survey respondents analyzed in this study. Study participants were recruited from individuals already engaging with the JEM chatbot. All surveys (overall evaluation and microintervention surveys) were open surveys, and a self-selection sampling method was used, representing a convenience sample. No direct contact with potential participants was made by any members of the research team. Recruitment occurred in writing through the chatbot’s “Provide Feedback/Feedback Survey” and “Skill Survey” conversation options, which users could select voluntarily. These options led to web-based surveys hosted by Qualtrics [41]. Surveys were administered with standard Qualtrics platform-level fraud-prevention settings applied to reduce duplicate responses (eg, cookie-based controls) [42,43], although these do not fully eliminate the possibility of repeat submissions in anonymous online settings. Only completed questionnaires were analyzed. Due to the fully anonymous design of the chatbot and surveys, individual users could not be tracked across sessions or timepoints. Therefore, engagement is reported at the session level, and survey participation represents voluntary, self-selected responses from the overall user base (see Multimedia Appendix 1 for aggregate study flow). For users who initiated the overall survey, completion rates were 80.1%, and for users who initiated microintervention surveys, completion rates ranged from 70.2% to 75.0%. There were no significant differences between the Australian and Canadian chatbot deployments for these completion rates (all Ps>.05).

JEM was a web-based and rule-based chatbot powered by the Google Dialogflow platform [23]. The chatbot was available 24 hours per day, 7 days a week to users in Australia via a Monash University website (Figure 1) and in Canada via the National Eating Disorder Information Centre website (Figure 2). These deployment characteristics are reported to contextualize accessibility and real-world use; however, the primary focus of the intervention is the structure and content of the psychoeducational and microintervention pathways delivered within the chatbot. The chatbot was free of charge to access and users could converse with the chatbot as long as they wished to do so. There was no login or authentication required to use JEM, and usage was completely anonymous. Users were cautioned on these websites that the JEM chatbot was not monitored by any humans, and if they were experiencing an emergency, they should contact country-specific emergency services with contact details provided.

The chatbot’s conversation content was based on evidence-based information or interventions for eating disorders, specifically psychoeducation, CBT, ACT, and mindfulness (for detailed information, see Beilharz et al [21]). Owing to the short conversational style of the JEM chatbot, the microinterventions taught by JEM were carefully selected, such as for CBT, education on cognitive distortions, for ACT, cognitive defusion exercises, and mindful breathing for mindfulness (see Figures 3 and 4 as examples). All microinterventions were estimated to take less than 5 minutes to complete. The chatbot operated through a rule-based decision-tree structure, where user inputs were matched to predefined intents that triggered specific conversational pathways. Users engaged with JEM via either free-text inputs or menu-based options, which routed them to psychoeducation content or presented a set of available microintervention scripts. The 10 microintervention scripts were discrete, standardized conversational pathways that users could actively select from a menu of available skills. Each microintervention followed a fixed sequence of prompts and responses with no within-script branching. These scripts were delivered in multimodal formats, including text, images, audio, and video. Users self-selected into microinterventions during their interaction with the chatbot, and each script could be completed within a single session or revisited multiple times. Completion was operationalized as reaching the end of the scripted conversational pathway.

There were no major conceptual differences in intervention content and technical delivery between the Australian and Canadian deployments. In terms of safety, natural language processing was used for JEM to detect user-typed risk phrases such as “I want to kill myself,” and “I wish I was dead,” and JEM responded with 24/7 crisis support services specific to Australia or Canada. Specifically, risk detection was implemented using Google Dialogflow’s intent classification and entity extraction [44] to identify free-text inputs indicative of self-harm and suicidality. When a high-risk intent or entity was detected, the system immediately triggered a predefined response flow that bypassed standard conversational pathways and directed users to country-specific 24/7 crisis support services. There was also a “Get Urgent Help Now” option for users to select where the chatbot responded with these same crisis support services. If a user was to type a prompt that the chatbot could not match to a preprogrammed answer, the chatbot responded with “I’m just a simple bot so I’m still learning how to respond to your typed message. Try using my buttons to have a conversation with me” along with crisis support service contacts. There were no adverse events reported by users during the study period.

The Statistical Package for Social Sciences (SPSS; version 31; IBM Corp) [54] was used for quantitative statistical analysis. Descriptive statistics were used to analyze demographic characteristics and user satisfaction and experiences. To explore if there were differences between countries, 2-tailed independent samples t tests were used for continuous data and Fisher’s exact tests for categorical data (2-tailed). To examine within-session pre-post microintervention changes in mood and body image, Cohen d was calculated as a measure of effect size [37], with 95% CIs reported. Analyses were conducted at the survey or session level due to the fully anonymous design of the chatbot, which did not allow the linkage of responses across users or timepoints. Any missing data were handled with listwise deletion, acknowledging that missingness in optional, anonymous web-based surveys may not be completely at random. Given the exploratory and hypothesis-generating nature of these analyses, no formal correction for multiple testing was applied [55]. Applying such corrections in this context may have increased the risk of type 2 error and obscured potentially meaningful signals warranting further investigation. The results were therefore interpreted cautiously, acknowledging the increased risk of type 1 error associated with multiple comparisons. The open-ended qualitative text responses addressing reasons for chatbot satisfaction ratings were analyzed using an abbreviated content analysis approach [56], given the brevity of the responses (30 words maximum). GS conducted initial coding, with HH independently reviewing and refining codes. All authors agreed on the final content interpretations.

Across all analyses, reported Ns refer to completed survey instances (survey responses). As shown in Table 1, from the overall evaluation surveys, respondents were, on average, in their early 20s, but age distribution varied substantially, particularly in the Australian sample. The respondents predominantly identified as girls or women, followed by nonbinary individuals. Around two-thirds of respondents did not identify as LGBTIQA+; however, when they did, queer was the most common identity. There were no differences between Australian and Canadian-based respondents for any of these demographic characteristics (all P values >.05). The majority of respondents were European or White in Canada, and around a quarter identified as culturally and linguistically diverse (CALD) in Australia. Owing to the different style of questioning required, ethnicity was not compared between countries.

As indicated in Table 2 from the overall evaluation surveys, the most common reasons in both countries for using JEM chatbot were that the respondents thought they may have an eating disorder, had issues finding information, and were simply curious about the chatbot. Respondents found the information they were looking for the vast majority of the time, and the majority intended to seek or maybe seek further support after using the chatbot. The overall satisfaction rating for the JEM chatbot was high on average, and the SUS rating was in the “excellent” category for both Australia and Canada [46]. There were no country-based significant differences for any of these findings (all P values >.05).

For the open-ended qualitative responses to reasons for the satisfaction ratings, this question was completed by a self-selected subset of respondents (Australia n=16; Canada n=12), which did not significantly differ between countries (all P values>.05). Given the lower response rates, responses were analyzed in aggregate rather than by country (Table 3). The response themes were mostly positive in nature, with the most common focused on JEM chatbot providing helpful information and being easy to use.

Four of the 10 total microinterventions included in the JEM chatbot’s conversation yielded sufficient completed survey responses (n≥34) from either country to conduct suitable within-session pre- or postchange analyses (Table 4). These 4 microinterventions were (1) Managing unhelpful thoughts (CBT and ACT strategies), (2) Managing emotions (ACT strategies), (3) Challenging beauty (CBT strategies), and (4) Feeling worthy (CBT strategies). Sample sizes for individual microintervention analyses varied and were modest for some subgroups; therefore, the findings should be interpreted with appropriate caution.

Across microinterventions, the within-session pre-post ratings were broadly similar in Australia and Canada, with effect sizes ranging from very small to large. The Managing unhelpful thoughts microintervention showed very small or small effect sizes across most measures, with moderate effects observed for anxiety. The Managing emotions microintervention showed the largest within-session effect sizes for depression, with moderate effects observed for body size satisfaction and the remaining measures showing very small-to-small effects. The Challenging beauty microintervention showed very small-to-moderate effects across measures, with confidence and physical attractiveness showing the largest effect sizes. The Feeling worthy microintervention showed very small-to-moderate effects across measures, with confidence, physical attractiveness, body size satisfaction, and body shape satisfaction showing the largest effect sizes.

This study provides one of the first real-world evaluations of a mental health–focused chatbot deployed at scale across 2 countries and continents, providing insight into chatbot use under naturalistic conditions. Overall, the findings indicated that the JEM chatbot was well accepted by respondents who completed surveys in both Australia and Canada, demonstrated excellent usability, and showed immediate within-session differences in mood and body image ratings. Satisfaction was high, information needs were largely met, and most survey respondents reported an intention to seek or consider further support after using the chatbot, suggesting that JEM may function as a supportive resource that could potentially facilitate consideration of further support.

Across microinterventions, patterns of within-session pre-post ratings were broadly similar in Australia and Canada, with effect sizes generally ranging from very small to large. While the magnitude of effect sizes varied across microinterventions and outcomes, the largest effects were typically observed for depression, anxiety, confidence, and body size satisfaction outcomes. These findings reflect immediate within-session pre-post changes and should not be interpreted as evidence of causal or sustained intervention effects.

In contrast, effect sizes for happiness ratings were generally very small across the microinterventions included in the analyses. This may reflect the brief nature of the interventions, which are designed to reduce distress and maladaptive cognitions rather than induce immediate positive affect, and is consistent with theoretical expectations of CBT and ACT informed strategies [57-59].

Gender identity beyond the binary was represented in the sample, with approximately 11% of respondents identifying as nonbinary. All study analyses included these respondents within the overall sample rather than stratifying by gender. Overall, JEM’s nongendered and inclusive framing of content [21] suggests potential applicability across gender identities, although future work should directly examine differential experiences among gender diverse users, particularly their responses to psychotherapeutic microinterventions.

These findings extend the very limited literature on eating disorder– and body image–specific intervention chatbots, particularly in real-world contexts. Previous RCTs have demonstrated efficacy for Tessa, Topity, and ED ESSI under controlled conditions [15,16,19], while other chatbots have been evaluated only partially or at early stages of development [17,18]. This study extends this work by demonstrating that a rule-based chatbot can be deployed with high user satisfaction and usability in naturalistic settings. Furthermore, the patterns of within-session pre-post differences observed among JEM users aligned in direction with findings reported in more controlled trials, while recognizing the differences in study design.

Compared with prior naturalistic evaluations of mental health chatbots such as Wysa [31], this study focused on a more specific population and set of outcomes. The patterns of within-session pre-post differences in anxiety, depression, and body image satisfaction observed among JEM survey respondents were mostly consistent with those reported in broader mental health chatbot evaluations. These observations are consistent with the literature suggesting that well-designed conversational agents may be associated with positive user-reported outcomes in real-world observational settings [31,60] and align with broader discussions about the promise and fragility of AI-mediated mental health care in real-world settings [61].

Importantly, the present findings contrast with concerns raised following the real-world deployment of the Tessa chatbot in 2023 in the United States, where the chatbot deviated from its rule-based design and provided dieting and weight loss advice [35]. No adverse events or harmful outputs were reported by users during the study period. JEM’s rule-based architecture and co-designed content were intended to minimize such risks; however, all safety outcomes were not assessed in this study.

The very small effects on happiness from microinterventions also align with previous digital intervention research, where reductions in negative symptoms are often more readily observed than increases in positive affect, particularly following brief, single-session interventions [59,62]. Together, these findings suggest that JEM performs in a manner consistent with existing evidence while addressing a notable gap in real-world evaluations of body image– and eating disorder–specific chatbots.

Recent RCTs of generative AI-based conversational agents further suggest both the promise and the complexity of chatbot-delivered mental health interventions, reinforcing the importance of safety-focused design and careful real-world evaluation [63]. While the vast majority of qualitative feedback in this study was positive, with respondents reporting helpfulness, ease of use, and an engaging conversational style, a minority had negative feedback. More broadly, rule-based chatbot systems are often perceived as less flexible than emerging large language model (LLM)–based approaches, which offer more naturalistic dialogue [64]. This highlights an important design trade-off between maintaining clinically safe, structured responses and meeting evolving user expectations for more adaptive and personalized conversational experiences. Future JEM iterations could explore hybrid architectures that combine rule-based safety constraints with controlled generative or retrieval-based components to enhance conversational flexibility while maintaining clinical safety and content fidelity [65].

Several limitations should be acknowledged for this study. First, the absence of a control group precludes causal inference and means changes cannot be definitively attributed to the chatbot alone. Second, outcomes were assessed immediately before and after individual microinterventions, limiting conclusions about the durability of effects. The immediate pre- and postintervention design may also be susceptible to demand characteristics and expectancy effects. However, differences were not uniform across outcomes or microinterventions, suggesting variability in within-session responses across measures and intervention content. Third, sample sizes varied across microinterventions, with 4 yielding sufficient data for analysis, which may have influenced the pattern of the findings based on differential engagement. Additionally, reliance on self-report measures and voluntary survey completion introduces potential response bias and means that the results reflect only a subset of users. Due to the fully anonymous design of JEM, which did not include login, authentication, or persistent user identifiers, individuals could not be tracked across sessions or over time. As a result, it was not possible to construct a participant-level CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) style flow diagram [66] tracking individuals through the chatbot content or determine precisely where attrition occurred within the chatbot journey. Engagement data are therefore reported via text and diagram at the session level, limiting the assessment of repeat usage and longitudinal engagement. Furthermore, this study did not assess changes in specific eating disorder symptoms, eating behaviors, or clinical outcomes.

Future research should examine longer-term outcomes associated with repeated or sustained chatbot use, including whether these immediate within-session differences are associated with longer-term outcomes, such as eating disorder risk or engagement with formal care. Hybrid designs combining real-world deployment with comparison conditions may help balance ecological validity with internal rigor. Further investigation into engagement patterns, personalization, and differential effects across demographic subgroups is also warranted.

This study demonstrated that a co-designed evidence-based chatbot, JEM, can be deployed at scale across multiple countries, achieving high usability and user satisfaction. Immediate within-session differences in mood and body image ratings were observed following the completion of the microinterventions investigated. However, further controlled research is needed, and the longer-term effects are yet to be determined. The study findings were broadly similar across Australia and Canada. These results extend the limited real-world evidence for mental health and body image– or eating disorder–specific chatbots, demonstrating that rule-based systems with co-designed content can be acceptable to users. As digital mental health interventions evolve, future research should explore how different approaches, including rule-based and generative AI chatbots, can complement each other to provide scalable, safe, and personalized support. Collectively, these findings suggest that chatbots such as JEM may represent a feasible and acceptable low-barrier resource for individuals seeking information and support related to body image and eating disorder concerns.