ITK was developed by and for adolescents aged 13-19 with a goal of increasing use of contraceptive and clinical health services [20]. Adolescents representing diverse priority populations engaged in a user-centered design process to help create the program’s content and digital components [21,22]. The curriculum is based on a positive youth development approach, which promotes personal strengths and healthy development through supportive opportunities and experiences [23,24].

The program was developed to be inclusive and to address the needs of homeless and unstably housed youth; youth of color; and lesbian, gay, bisexual, transgender, and queer or questioning (LGBTQ+) youth. ITK combines 6 hours of in-person sexual health education with technology-based content to provide the skills, information, and resources necessary to improve the sexual and reproductive health and overall well-being of adolescents.

The intervention is divided into three modules: (1) sexual health and contraceptive use; (2) healthy relationships; and (3) educational and career success. Health educators incorporated different technology-based components in each module, such as videos, online goal setting, career opportunities, and geo-location of local services. Some content was “gamified” using Kahoot, a game-based learning platform, and app-based quizzes and activities to earn points. Health educators concluded each module with a guided activity on the app and then assigned a task for the youth to complete outside of class. Youth could also sign up to receive text message reminders of key content and personal goals. These tools as well as additional resources and quizzes were available through a downloadable app or website, enabling youth to access the information outside of the in-person sessions. Health educators provided tablets with the app previously installed for use during the in-person sessions, though the participants also were encouraged to download the app on their mobile phones. The health educators helped to troubleshoot any technical issues youth were experiencing with the app.

The health educators received training on the curriculum, classroom management, and the technological components prior to implementation. This included in-person trainings and “teach backs” as well as shorter refresher trainings throughout the implementation period. The health educators reported technical issues about the app to the website developers. In addition, the developers updated features and replaced broken links over the course of the program. The researchers, health educators, and the app developers met biweekly to discuss any implementation challenges and adaptations.

ITK was implemented in 51 cohorts (groups) with 559 youth at 36 youth-serving agencies representing a variety of settings where youth receive services or activities in Fresno County, California. The health educators traveled to the sites of the participating agencies for implementation, which included school and after-school settings, employment and training sites, youth development centers, clubs, foster care sites, housing authorities, tribal agencies, and LGBTQ+ programs. The majority of participating youth were Latino (70% [n=381]) with a mean age of 15.5 years (SD=2.07). Almost all of the participants owned or shared a smartphone (89% [n=480]), and 86% (n=469) had access to the internet in their homes.

Over the 3 years of implementation, a total of 6 health educators implemented ITK, with an average of 3 health educators per year. The health educators had a range of educational backgrounds, prior teaching, or training experience, and were comfortable with technology. This varied from 1 educator with over 6 years teaching comprehensive sexual health education to 2 educators with no prior experience in sexual or reproductive health and limited familiarity with technology; 2 other health educators had at least 2 years of experience implementing sexual health education in similar settings. Moreover, 2 health educators were male, and all lived in Fresno County.

This implementation study used data collected as part of the cluster randomized controlled trial [20]. Due to the complexity of the intervention being evaluated, a better understanding of the contextual factors, including the technology and in-person implementation, can help to improve future interventions and interpret the intervention’s outcomes [25]. Process data from implementation logs and annual interviews with health educators were collected to assess fidelity to the intervention and to promote ongoing quality improvement.

This study used a modified form of grounded theory in which a set of potential concepts were identified and coded, and additional themes were inductively identified from the data [26]. The qualitative analysis was guided by structural themes based on key areas of research interest, such as technology use, emerging themes from the review of transcripts, and the open-ended responses in the implementation logs [27]. This mixed coding system combined an initial list of codes using the main research questions and additional codes that were added based on further review [28].

One researcher coded all transcripts, while another double-coded a subset and reviewed coding for intercoder consistency. The coded interviews had an interreliability score of 0.80. The research team met regularly to review the coding process, clarify codes, and update the codebook. As needed, the researchers reviewed the quotes that were coded differently and jointly agreed to their coding. The codes were analyzed for patterns, with relevant themes extracted. The findings were also compared by year and by health educator to assess if experiences varied over time or by person. The qualitative coding was conducted using Dedoose, version 8.0.35 (SocioCultural Research Consultants, LLC) [29].

The responses to the closed-ended question regarding the frequency of technology-related interruptions were extracted and summarized using Stata 16 (Stata Corp). We used the Fisher exact test to compare the responses by whether the cohort received the program in the first year of implementation and in a school setting. One-sided P values are reported.

Implementation logs were completed for all 51 in-person sessions of ITK conducted between October 2017 and February 2020. During this time, 8 interviews were completed with the health educators. During the first year of implementation, the health educators reported that technology issues affected implementation to some degree of the time in 7 out of 8 cohorts (87%) with that amount decreasing over the next 2 years, to 11 cohorts out of 19 (58%) and 7 cohorts out of 15 (47%), respectively (Figure 1; note that there were missing responses from 9 implementation logs in year 1 since one question on technology issues was added later). When calculated with the Fisher exact test, the difference between the first year and subsequent years was only marginally significant (P=.08) due to the small sample size. The cohorts implemented in non–school settings such as in group homes or community-based organizations were much more likely to have technology issues than those in school settings; 14 out of the 19 (74%) non–school setting cohorts experienced technology issues compared to 11 out of 23 (48%) of cohorts implemented in a school setting (P=.05).

Successes and challenges emerged in 3 key domains: managing technology, usability of the ITK app, and youth engagement. Managing technology included issues related to meeting the technological requirements and administrative needs for implementation during the in-person ITK sessions, such as device compatibility, internet access, and availability of necessary technology hardware. The topics related to the ITK app’s usability were those specific to the content and functionality of the app, the integration of the app into the in-person ITK curriculum, and the participants’ use of the app. Youth engagement referred to how integrating technology into the curriculum affected the participants’ focus and engagement during in-person implementation. Note that many of the issues overlapped; for example, challenges with internet connectivity limited access to the app’s content, which then affected youth engagement. Table 1 summarizes the successes and challenges within these 3 domains.

Implementation of ITK involved managing multiple technological devices and administrative requirements such as connecting program tablets and participant smartphones to the internet and projecting digital content on a screen. The health educators noted that implementation was easier in sites that had the necessary audiovisual projecting devices, such as a monitor and projector and internet access. The health educators consistently reported the challenges associated with using technology during implementation, though this decreased in frequency each year. An inability to access high-speed wireless internet was the most commonly described technology issue reported. The health educators adapted to this issue by bringing their own mobile Wi-Fi hotspots to the sites. Other common technology issues included tablets freezing or crashing during use, the lack of audiovisual projecting devices, broken web links to external online content, and youth forgetting log-in information. One health educator described common experiences with the technology:

These technology-related issues caused delays and required health educators to adapt how they delivered the program, both ad hoc and while preparing for future implementation sessions. One health educator described an example of an ad hoc adaptation as such:

The health educators provided tablets for participants to access the app if they did not download it on their personal phones. While this increased access to the materials, managing the tablets required significant planning and preparation time, as health educators needed to ensure that all electronic devices were charged and functioning properly. One health educator gave the following explanation:

The health educators generally had positive comments about the content of the ITK app, particularly the interactive map linking youth to resources within their community. The health educators used these online resources with the participants to identify community clinics, counseling services, and help lines for youth and families experiencing violence. One health educator described the benefits of having resources consolidated on the app:

While youth could access the ITK website on a tablet during class, ITK originally was designed as an app for youth to download on their phones for later access. However, youth often faced challenges in using the mobile app including limited storage or data, limited battery life, lack of a personal cell phone number, difficulty remembering the required password, and sharing the phone with other family members. One health educator noted the following experience:

Youth also expressed reluctance to download the app due to confusion about the purpose and utility of the app as well as its connection to the in-person curriculum, resulting in a limited use of the app outside of the in-person sessions. Additionally, some youth did not have access to a smartphone at all, which not only prevented them from accessing the app outside of the in-person sessions, but also contributed to the participants feeling left out of the program. A health educator described the experience of 1 youth who was homeless as such:

Health educators also noted a lack of integration between the in-person elements of the curriculum and the ITK app. Because many of the ITK app features and activities were explained at the end of the modules, health educators commented on the difficulties of transitioning between the in-person curriculum and technology-based activities. One health educator stated the following:

Overall, the health educators reported that youth were engaged and interested in the curriculum. They stated that participation and engagement increased among the youth when playing games with Kahoot, an online learning platform that allowed educators to gamify content delivery. One health educator explained it as such:

Youth also responded particularly well to activities utilizing the O*Net OnLine website, a free online career exploration tool. However, health educators also noted that youth preferred participatory activities in general, whether technology-based activities and games or in-person activities such as role plays compared to lecture-based activities. One health educator described their experience as follows:

Despite fostering interest and engagement, in some instances, the presence of electronic devices was distracting for some youth. One health educator described a common experience in an implementation log as follows:

Another common youth engagement issue was the need to contextualize or personalize content for the participants. On almost every implementation log, the health educators noted instances where they had to reframe content or add explanations. For example, 1 health educator noted their role in providing supplemental information regarding a video on the biology of conception and pregnancy:

These findings illustrate some of the successes and challenges of integrating digital technology into an in-person sexual health education program from the critical perspective of health educators. As previous studies found, health educators commonly reported that technological issues such as connectivity and device compatibility affected implementation, which were not unique to sexual health education [9,17,18]. However, technological issues became less frequent over time, likely for 3 reasons. First, health educators gained experience and confidence in addressing common technological challenges, including making innovative adaptations or finding alternatives when technology malfunctioned. Second, additional training may have led to greater familiarity and comfort with the myriad of platforms and implementation strategies. Third, health educators provided ongoing feedback to the developers, which resulted in changes to certain technology features and problem resolution. The decline in technological issues demonstrates the importance of ongoing and iterative quality improvement processes and the need for sustained engagement by the app development team in any technology-based health education intervention. It also illustrates the need to ensure that health educators are comfortable and confident in using technology, either through prior experience or through training.

Despite the implementation challenges, the health educators held positive views about the value that technology added to the in-person education, particularly in engaging youth with the material. Overall, youth tended to be more involved when they actively interacted with the content, whether through the technology-based components or in-person activities. Technology may be one of many tools that can increase the interactivity of curricular content [12]. A review of a variety of computer-based technologies found that digital games had the most evidence supporting their use to increase student engagement [30]. Game-based activities were successful, supporting the evidence that well-designed gamification can increase student engagement and motivation, and demonstrating the potential for gamification of educational content [31,32]. While the digital content was generally well received by youth, health educators also noted that the technology-based activities were not fully integrated into the curriculum. This was similar to the findings by another study of an online sexual health education course where some teachers reported difficulties transitioning between web-based and in-person activities [9].

Although adolescents have widely adopted technology, our findings are reflective of research showing ongoing disparities in technology access and use at the individual, community, and institutional level [7,33]. While ITK was designed for youth in underserved settings including foster care and shelters, health educators were more likely to encounter technology issues such as lack of Wi-Fi and other hardware in non–school settings. This made the implementation of the technology components of the program more challenging [34]. Additionally, while most participants had phones, some had limited storage or shared the phone with other family members, making them less inclined to download or keep an app, particularly one that stored sensitive information. By contrast, other studies have found that youth appreciate the anonymity available through technology-based sexual health interventions [34].

While technology can enhance youth engagement and comprehension, this study highlighted the critical role of health educators who secure the hardware necessary for implementation, adapt the curriculum when technology fails, and contextualize and personalize digital content to meet the unique needs of the youth they serve. Other studies have demonstrated the importance of staff training, confidence, and self-efficacy for the success of efforts to integrate mobile technology into education [35,36]. Beyond technological competence, health educators also need the core capabilities in knowledge and skills to deliver effective, inclusive, and appropriate sexual health education, particularly when discussing sensitive sexual and reproductive health topics [37].

A few limitations should be noted. The implementation log data is self-reported, so health educators may have underreported issues or interpreted a situation differently. However, these results also were consistent with annual interviews with the health educators. This study did not assess the prior experience or comfort level of the health educators with technology. In addition, because the ITK app changed over time in response to feedback and updates, some of the technical components or issues may have been resolved over time or varied by time period.

As more sexual health educational programs incorporate technology, they should consider the specific role and use of technological components from both a pedagogical and logistical standpoint. Developers should engage with youth and health educators when designing health curricula and apps to ensure that the content is integrated and promotes youth learning and engagement. App developers need to invest in usability testing and a system for reporting issues throughout implementation and iteratively update the product based on that feedback. Similarly, developers and organizations need to ensure that health educators have the training, confidence, and support necessary for successful implementation, including the curricular content, classroom management skills, and necessary technology.

Although technology is often presented as a solution to reach underserved populations, that premise is not yet fully realized. Educational programs considering the adoption or integration of technology should assess the potential needs and technological capacity of the participants and settings.