Although pediatricians frequently discuss behavioral health concerns with families [1], they report a lack of training on how to effectively address these concerns using evidence-based approaches and collaborative communication [2,3]. This represents a critical training gap, as the implementation of effective behavioral management strategies by parents to address typical childhood behaviors (eg, tantrums) can mitigate the development of future behavioral health disorders [4]. With approximately 20% of children currently having a diagnosis of a behavioral health disorder [5], identification of training strategies to support future pediatricians’ skills in delivering behavioral health anticipatory guidance (BHAG) is needed [6]. Recent curricula aiming to support BHAG skills in pediatric residents have primarily relied upon experts to deliver content via didactic or case-based discussions, limiting the opportunity for spread and scale of effective interventions [7,8]. Simulation-based medical education in the form of patient actors has been incorporated into training pediatric residents on BHAG; however, limitations related to accessibility, realism, and psychological safety have been reported [9]. Advances in technology offer an opportunity for broad dissemination of theory-based education. Screen-based virtual reality (VR) is one technology that can be leveraged to support BHAG skill development in pediatric residents to optimize pediatric health outcomes.

We previously developed a novel screen-based VR curriculum to enhance BHAG competencies among pediatric residents through deliberate practice of skills [10-12]. A human facilitator drove both the VR responses of graphical characters in scenarios and provided personalized performance feedback following each simulated scenario. The VR curriculum proved highly acceptable and efficacious in enhancing BHAG and health care communication skills [10,12-14]. During usability testing of the human-facilitated curriculum, residents reported the curriculum as realistic, engaging, practical, appropriately scaffolded, and psychologically safe. They indicated the curriculum was easy to use and reported high levels of immersion and spatial presence within the virtual environment [10]. The spread and scale of this education was limited due to the need for highly trained human facilitators. Artificial intelligence (AI) offers an opportunity to explore how automation may support maintenance and scale of such VR-based educational interventions.

AI allows users to interact with computer systems that use data sources to interpret user inputs and deliver curated responses [15]. The use of AI in medical education and simulation has been limited; prior research indicates limitations related to realism and accuracy [16,17]. Thus, we sought to assess the feasibility and acceptability of an automated curriculum to support BHAG and communication skills in pediatric residents.

This study was reviewed and determined to be exempt by the Cincinnati Children's Hospital Medical Center Institutional Review Board (2023-0314). A waiver of documentation of informed consent was granted; therefore, prior to data collection, participants reviewed a study information sheet, and participation in study procedures indicated their consent. All study team members adhered to institutional procedures governing data access and privacy. Compensation was provided to all participants.

To develop the automated intervention, we partnered with BreakAway Ltd, a game developer that created a conversational interaction system titled Standard Patient Studio (SPS) [18]. SPS uses a cloud-based AI platform in which progression through a simulated scenario is driven by a user’s verbalizations. A multidisciplinary team that included 2 pediatricians, 1 pediatric psychologist, 3 software developers with expertise in AI, and 2 research assistants adapted the human-facilitated VR curriculum to the SPS platform. Procedures included scenario storyboarding with branching logic using the prior human-facilitated VR scenarios as a guide, with iterative scenario development testing by study team members. The final curricular prototype yielded 2 scenarios that replicated the human-facilitated VR curriculum’s learning objectives and used identical verbal statements by the graphical parent. Potential answer options for the learner were displayed on the screen during each scenario to support scaffolded learning and to inform a learner’s spontaneous verbalizations, thereby ensuring progression through a scenario. The answer option most closely aligned with the user’s verbalizations was selected by the natural language understanding system. This system categorizes user input via a medical taxonomy reference that assigns an appropriate patient action. In prior trials, this system has demonstrated a 92% appropriate response rate [19]. On the basis of the selected answer option, the user received immediate feedback (ie, excellent, acceptable, or poor response) via color-coded emojis that were previously embedded in the SPS platform. Upon scenario completion, users received feedback indicating why an answer selection was preferred or not preferred. The feedback mirrored the human-facilitated feedback by identifying the specific BHAG and motivational interviewing skills demonstrated during a scenario and where there were missed opportunities to demonstrate such skills. A low performance score required users to repeat the scenario, which mirrored the deliberate practice approach used for the human-facilitated VR training.

Usability testing, a critical component of intervention development, which seeks to systematically evaluate the extent to which an intervention achieves its intended purpose by engaging experts to identify weaknesses, occurred using mixed methods [20,21]. Health care team members, including senior faculty, psychologists, simulation educators, and pediatric residents, were purposefully sampled to pilot the curriculum. Usability testing occurred in person, where users were asked to “think aloud,” verbalizing their perspectives as they navigated the automated curriculum [22]. Following completion, participants underwent a semistructured interview to explore their overall impressions (Textbox 1). Interviews were recorded, transcribed, and analyzed by 2 raters (AM and FR) using the rigorous and accelerated data reduction (RADAR) technique, a rapid qualitative analysis approach [23]. RADAR uses a 5-step approach to data reduction through the iterative refinement of data tables and is well suited for studies with narrow research questions. The raters ensured consensus on reduction decisions at each step. Trustworthiness of the qualitative data was also supported through the use of an interview guide that generated descriptive responses (credibility), multiple analyzers and peer debriefing (dependability), and recording of detailed procedures (confirmability) [24]. Participants also completed the System Usability Scale (SUS) [25] to examine the platform’s ease of use and a subset of items from the Measurement, Effects, Conditions: Spatial Presence Questionnaire (MEC-SPQ) [26] to assess for immersion in the virtual environment.

Of the 9 participants, 8 (89%) provided qualitative data. Participants described the scenarios as relevant and realistic. Overall, they reported that the system was easy to use and correctly coded their verbalizations. The study team observed, and participants reported, that there was a learning curve associated with using the online microphone. Participants indicated appreciation for the nuanced answer options and level of difficulty, although some participants desired decreased scaffolding and increased free agency. Some participants described a game-based experience, attributed to the immediate feedback in the form of color-differentiated emojis. Some viewed the game mechanics integration positively, while others felt it detracted from realism. All participants (n=8) indicated that they would recommend the curriculum for learners. Due to the inclusion of game-based elements and the select option format, some participants believed the curriculum was best suited for novice learners (eg, medical students and residents) rather than practicing pediatricians. All participants indicated that the feedback accurately reflected their performance. When asked what one change participants would make to the automated curriculum, 3 (38%) requested more free agency and less scaffolding during the simulations, 2 (25%) wanted shorter answer options, 2 (25%) wanted more variability in terms of graphical character responses, and 1 (13%) wanted more specific feedback regarding incorrect answer selections.

Table 1 includes exemplar quotes informing these data interpretations.

The overall SUS mean score among participants was 78 (SD 9.8; range 65.0-92.5), indicating good to excellent usability of the system [27]. On the MEC-SPQ, 44% (4/9) of participants agreed or strongly agreed that it felt like they actually participated in the action of the simulation. Only 11% (1/9) agreed or strongly agreed that they felt as though they were physically present in the environment. (Table 2)

Adaptation of a human-facilitated VR curriculum on BHAG was successfully automated through the use of the SPS platform developed by BreakAway Ltd. Qualitative and quantitative data indicated that users could easily navigate the platform and interact with its components. Compared with the human-facilitated VR curriculum [10], users reported less spatial presence in the virtual environment. However, users still described the automated training as realistic and relevant. Users also endorsed a game-based aspect to the training experience, which was not reported with the prior human-facilitated VR curriculum [10]. Overall, users indicated that the current automated training might be most appropriate for novice learners. Our decision to scaffold the learning experience by displaying answer options supported reliable progression through the scenarios and safeguarded against misclassification of verbalizations, which can occur when using generative AI models [17,28-30]. However, more advanced learners may benefit from removing scaffolded answer options at a limited number of interaction points.

This study had several limitations. First, our small, purposeful sample, ideal for usability testing [31-33] and intervention development, limited generalizability. However, recent literature by Guest et al [31,32] indicates that 6 to 7 interviews are likely sufficient to support theme generation for homogenous samples such as ours. Second, the rapid qualitative analysis method may have limited exploration of interview data. However, the RADAR technique is well suited for research questions that are narrow in scope [23].

Behavioral health medical education lacks effective, innovative training curricula necessary to develop practice competencies [3]. By removing the need for a human facilitator or patient actors, AI-based communication training provides an opportunity for spread and scale [16,34]. Additional research is needed on how to best incorporate AI into education interventions to optimize learning outcomes that support behavior change. We specifically sought to align the automated curriculum with deliberate practice principles, as this served as the theoretical foundation for the prior human-facilitated VR curriculum [11]. Deliberate practice is a personal, goal-oriented approach to training that focuses on attempting a behavior (eg, providing BHAG), receiving immediate feedback, and then repeating it until demonstrating skill mastery. Consistent with this theoretical framework, the automated curriculum provided a platform for rehearsal of specific verbiage, feedback based on each interaction point, repetition of skills over the two scenarios, and the opportunity to repeat scenarios. Aligning AI-based educational interventions with evidence-based adult learning theories will allow us to establish a literature base on how AI might most effectively support learning. This is particularly important for the topic of behavioral health, as recent studies describing curricular interventions often do not specifically indicate a theoretical foundation for the work [8,35,36]. As AI becomes integrated routinely into training, we should consider how we might best facilitate trust for learners interacting with AI systems as well as provide transparency regarding AI algorithms and source data [37].

Our study demonstrated the feasibility of developing an automated curriculum to support BHAG skills in pediatric residents through collaboration with an experienced industry partner. Automation of training curricula adds value by enhancing the capacity for scalability by reducing dependence on human facilitation. As such, automation also decreases the costs for participation, supporting equity among learners in accessing novel, evidence-based education. Next steps include updating the curriculum based on usability testing and implementing it among pediatric residents to assess its acceptability and effectiveness among that population and to explore how its outcomes compare to other training modalities, such as human-facilitated VR interventions. An evaluation of implementation strategies will also be critical to understand how such novel education can be integrated into resident curricula. Moreover, given the varying perceptions related to the game-based elements incorporated into the intervention, further exploration of how such game-based features impact attitudes and learning outcomes is warranted. We also plan to add an orientation regarding learning goals prior to intervention completion to align with best practices for promoting a constructive digital feedback environment [38]. In the future, we aim to refine the automated curriculum, including adjusting the level of free agency and gamified elements, to support learning across the continuum of health care providers.