A Chatbot-Based Coaching Intervention for Adolescents to Promote Life Skills: Pilot Study

Introduction

Background

Adolescence is a challenging period characterized by rapid changes [1] and increasing emotional and social demands. Interventions that foster life skills, coping, and well-being are particularly important during adolescence; virtual coaching solutions could greatly enhance delivery of these interventions in both school and out-of-school settings. The aim of this study was to co-design a life skills coaching intervention with adolescents to be delivered by a chatbot; this would be done by conducting an initial participatory design workshop followed by testing the feasibility of the whole intervention with another convenience sample of adolescents. The life skills virtual coaching intervention was meant to be used either in out-of-school settings for individual training on life skills or in combination with school interventions on life skills provided by a human coach (ie, teacher or domain expert).

Literature Review

Several challenges and emotional demands characterize adolescence, often affecting the mental well-being of youths. Among these, bullying and cyberbullying is recognized nowadays as a major social problem, affecting 37% of adolescents [2,3], with extensive negative consequences for the victims involved. Research has shown that adolescent bully victimization is associated with poorer school achievement [4]; lower self-esteem; and increased loneliness, depression, and anxiety [5], whose consequences persist into adulthood [6]. Interventions that foster life skills [7], coping, and well-being are particularly important during adolescence as protective and preventive strategies against the consequences of bullying and cyberbullying. Life skills include the ability to exhibit adaptive and positive behaviors that enable individuals to deal effectively with the demands, challenges, and stress of daily life [8]. Childhood and adolescence are the developmental stages during which one acquires these skills through various methods and people [9]; a positive technology approach [10,11] can support the design of such experiences in a digital format. These solutions have the potential to be highly scalable, since almost three-quarters of adolescents (73%) have a mobile phone or have access to one [12]; as well, digital interventions have been proven to be effective at changing a range of health behaviors [13]. Digital assistants or chatbots are conversational agents that can be easily used to support the delivery of educational interventions for mental health and well-being [14]. A chatbot conducts an interaction through conversation with its users by simulating humans’ dialogue patterns and behaviors. However, there is still a paucity of research showing the design and effective usage of chatbot interventions for supporting mental well-being of adolescents [15].

Methods

Phase 1: Co-Design Workshop and Participants

The co-design phase of the study involved a sample of 20 adolescents (age range 14-15 years) attending a first-year class of a secondary school in Northeast Italy. A participatory design workshop was organized and led by a research staff of four psychologists. The staff involved the students in using and commenting upon a prototyped session of the chatbot intervention to collect their needs and preferences on the following: the chatbot’s look and feel, the type of content and duration of the session, their unmet expectations regarding the prototype, and suggested improvements. Participants provided signed parental consent. Discussions held during the initial co-design workshop were audio-recorded and transcribed. Qualitative data collected were analyzed using a modified version of Braun and Clarke’s guidelines [16] for thematic analysis. Specifically, two members of the research team read the transcripts and independently developed categories of responses. Agreement on the proposed categories was reached through discussion.

Based on the co-design results, the research staff developed the entire life skills coaching intervention, including eight online sessions, delivered by means of a series of coaching dialogues, exercises, and video cartoons presented by the CRI (for girls) and CRIS (for boys) chatbot—the chatbot was named CRI and CRIS as abbreviated forms of the Italian names Cristina (female) and Cristiano (male), respectively. As shown in Figure 1, each session starts by chatting with the CRI or CRIS chatbot, which looks like an older peer—boy or girl—of the user. The chatbot asks the user to watch a 1-2-minute video cartoon about a bullying or cyberbullying situation involving adolescents, which is relevant to the skill addressed, embedded within the conversation system. The chatbot then continues the conversation by asking questions and proposing exercises to trigger user self-reflection on how to cope and behave in such a situation. By the end of the session, the chatbot asks the user to watch another video cartoon where the CRI or CRIS coach provides advice on constructive strategies for coping well with the challenges and emotions triggered by the situation (see Multimedia Appendix 1 for an example of the coaching session). Each session was designed to stimulate self-reflection on life skills and coping strategies relevant to bullying or cyberbullying situations [9], including emotional self-awareness, social awareness, interpersonal relationships, conflict resolution, assertive communication, sadness and loneliness, leadership, and positive emotions. Each session lasted between 5 and 15 minutes. To facilitate engagement, the coaching sessions were personalized according to gender; the chatbot represented a peer of the same gender—girl or boy—as the participant (see Table 1).

‎View this figure

Phase 2: Feasibility Test Participants and Setting

A convenience sample of 21 adolescents (13/21, 62% male; 8/21, 38% female) in the age range of 12-17 years (mean 14.52) were invited to use the entire coaching intervention over 4 weeks, participating in two sessions per week. The aims of this formative and qualitative study were as follows: (1) to assess the perceived value of the coaching intervention for a population of adolescents having a wider age range with respect to the group involved in the co-design workshop and (2) to check the user experience with the full set of online coaching sessions in order to refine and finalize their content. Due to the focus of our study—assessing human factors of the virtual coaching experience provided—and the homogeneity of our target group, a sample size of around 20 adolescents was considered appropriate [17,18].

After providing signed parental consent, participants were instructed to access the chatbot from any preferred device by visiting a website where the coaching sessions had been published for the formative evaluation. An anonymous 10-item satisfaction questionnaire was delivered online at the end of each coaching session; a 5-point Likert scale was used to rate overall usefulness, ease of use, and the value of the program, and suggestions for improvements were provided via open-ended questions. Descriptive statistics were calculated for satisfaction items.

Results

Phase 1

Regarding the interaction with the prototyped session in the co-design phase, three main themes emerged. First, participants found the educational video cartoons provided in the session to be appealing and helpful in triggering self-reflection on the challenging situation presented: “The black and white video presenting a real-life setting helped me to reflect more on myself, how I would react in that case ...” Second, participants reported that the need to type answers during the chat when replying to the chatbot’s questions helped them to reflect on their feelings and thoughts relevant to the topic, even if they knew that the chatbot was not able to understand their replies as a human being: “Even if I know that the chatbot cannot reply as a real person, since it is not able to understand everything, the fact of typing in the chat what I think about the situation helps me to reflect ...” Third, participants said that the chatbot looked like a nice, smart old friend, someone you can trust and talk with when you want to address important issues: “I like this chatbot, he looks like a boy, just a bit older than me, but not an adult; he is sensitive and very smart.”

With respect to potential improvements, participants found that some parts of the dialogue with the chatbot were a bit unclear or were missing empathy from the chatbot side: “I got confused; after a video, I don’t know what I should say to go on with the session” and “It would be great if the chatbot would reply with something more relevant and empathic to my comment.” Second, some participants reported that parts of the session were too redundant or the chatbot provided too much text that required scrolling up the chat window: “... at some points, the bot gets a bit repetitive, it repeats the same thing many times” and “When the bot sends too much text in a turn, I need to scroll up not to miss the first lines and this is a bit annoying.” Supporting quotes are displayed in Table 2.

Phase 2

All 21 participants involved in the feasibility test completed the eight coaching sessions and the relevant questionnaires. For all eight sessions, on a scale of 1 (very little) to 5 (very much), the majority of participants (16/21, 76%) gave a rating of 4 or 5 in response to the question “How useful was this session for you?” Ratings were also 4 or 5 for ease of use (19/21, 90%) and innovativeness (17/21, 81%) of sessions. Participants' ratings on usefulness and ease of use for each specific session are reported in Table 3. Note that all sessions have a mean rating above 3, corresponding to the positive side of the scale, for both usefulness and ease of use.

Regarding the design of the contents provided in the sessions, 71% (15/21) of participants reported a preference for the coaching videos and 66% (14/21) expressed a preference for having each session last between 5 and 10 minutes. Among the possible improvements, participants mentioned a wider range of answer options to some of the chatbot questions and suggested removing redundancy in the dialogue of the longer sessions. Regarding perceived value, 95% (20/21) of the participants indicated they would recommend the CRI or CRIS chatbot to a friend.

Discussion

Principal Findings

To our knowledge, today there is still a paucity of chatbot-based interventions for life skills training and well-being promotion among adolescents, as well as a lack of data and guidelines on how to effectively design the user experience with educational chatbots for this intervention domain. However, use of Internet and digital solutions by youth is increasing worldwide, providing cost-effective opportunities for reaching this population with the delivery of self-help educational programs. Also, recent studies support our findings, showing that digital interventions based on chatbots or videogames can be highly engaging, improve well-being, and reduce stress for this population and for nonclinical populations [19-21].

The chatbot intervention that was co-designed and evaluated in this study was rather easy and fast to implement and was well-received by adolescents. The two phases of our participatory design process took into account the feedback and suggestions of an overall sample of 41 adolescents; the results reported can guide the future development of virtual coaching solutions for adolescents’ training, solutions that have been found to be acceptable and appealing to use for this target user group. The deployment of educational video cartoons and dialogue-based interaction with the chatbot turned out to be very engaging and useful for self-reflecting on the challenging situations presented, making the virtual coaching sessions a very promising digital environment for experiential forms of learning in youth. The virtual coaching experience that was designed could easily be integrated into existing school programs and interventions for bullying and cyberbullying prevention; this coaching experience could also be considered a practical way of providing out-of-school interventions for life skills training.

This study is limited by the participant sample size and by having been conducted only in Northeast Italy, which impacts the generalizability of our results to other countries and settings. However, mental health promotion programs for adolescents provided by a variety of delivery platforms, including digital media, are strongly recommended by public health policies in many countries and by the World Health Organization [22]. These recommendations support the international relevance of our study.

Conclusions

In summary, this study presented the co-design and formative evaluation of a chatbot-based coaching intervention for adolescents’ life skills training, which was grounded in a positive technology approach and was well-received by adolescents. Further research is needed for a more in-depth evaluation of the efficacy of this intervention in strengthening coping strategies and resilience.

A future step of this study consists of integrating our chatbot intervention on life skills into the Cyberbullying Effects Prevention (CREEP) platform for cyberbullying prevention, which will be tested for efficacy by involving approximately 200 secondary school students in Italy and France.