We recruited two participant groups in urban US communities: (1) individuals aged 18-28 years; and (2) health workers, who were individuals contracted or employed by health departments to encourage the uptake of COVID-19 vaccines. We posted ads on Craigslist and Twitter targeting young people and health workers in Baltimore, Charlotte, New York City, Philadelphia, and Washington, D.C., and we used snowball sampling through professional contacts to identify health workers. We sought to achieve variability along lines of race and ethnicity to represent our targeted users. For both groups, we excluded people who stated they would “definitely NOT choose to get a COVID-19 vaccine by August 2021” in a scaled response, since the chatbot aimed to target users along the vaccine hesitancy continuum excepting those refusing vaccines [42-44].

Users were invited to participate in 3 possible activities—a web-based questionnaire, Zoom-based interview, or a web-based focus group discussion—described elsewhere [30]. Participants were given US $20 Amazon e-gift cards for each study activity completed.

This formative study was approved by the Johns Hopkins Bloomberg School of Public Health Institutional Review Board (protocol number 15714).

We assessed the functionality or ease of navigation with the CUQ. Questionnaire results, displayed in Table 2, indicate that most participants agreed with the statement that the chatbot was “easy to navigate” (51/58, 88%) and “easy to use” (50/58, 86%), with a corresponding proportion disagreeing that it was “very complex” (47/58, 81%). Half (29/58, 50%) of the questionnaire respondents agreed that the chatbot “understood me well,” and 74% (42/57) disagreed that it would be “easy to get confused when using the chatbot.” Additionally, 91% (53/58) of respondents disagreed that “the chatbot seemed unfriendly” but only half (32/58, 55%) felt the personality was realistic and engaging. Finally, 43% (25/58) disagreed with the statement that “the chatbot seemed too robotic.”

CUQ scores, normalized out of 100, were calculated for 56 of the 58 participants; 2 participants did not complete all 16 questions within the questionnaire. The mean CUQ score was 70.2 (SD 12.1) with a median score of 70.3 (range 40.6-95.3).

Interview participants described four contributing factors necessary to achieve usability, as shown in Figure 2: (1) functionality, or ease of use/navigation; (2) performance, or the chatbot’s ability to understand and accurately respond to queries; (3) response efficiency and quality; and (4) confidentiality and privacy of the tool. Participants described two primary contributing factors to achieving credibility: (1) institutional credibility and (2) chatbot persona.

Interview participants listed institutional credibility and chatbot persona as key factors contributing to their determination of the chatbot’s credibility. Their assessment encompassed judgments around the chatbot’s origin, how reliable the content appeared, how they felt about using it at this stage of the pandemic, and how website design influenced their decision to use the chatbot.

Young people said the tool would help in discussions with vaccine-hesitant friends, family members, or members of their community. In all, 9 (16%) out of 58 questionnaire respondents described using the chatbot to encourage others to get vaccinated (see exemplar quotes in Table 3).

The 6 interview participants who served as health workers felt the chatbot could support their work. First, they must keep on top of emerging concerns in the community and look up new questions. Second, listening one on one to individuals was an important part of their role. As one middle-aged health worker (IDI02) said,

Participants described addressing the public’s concerns about vaccines via phone, for instance, in contact tracing or at health fairs, with many queries collapsing into a batch of common questions. As one young female health worker (IDI09) said:

Health workers noted the potential of a chatbot as a source to easily access up-to-date content. As one young health worker (IDI01) said, once the resource was approved by her department of health, “I’d be using it like every time I don’t know an answer or honestly...just to double-check my work.”

This study took a mixed methods approach to measure the perceived usability and credibility of a COVID-19 vaccine information chatbot with natural language processing capability in a web-based chat environment. An ethnically and racially diverse sample of urban-dwelling young people and health workers assessed the chatbot as achieving high usability in that it was easy to use, performed well in understanding their inputs, and offered advantages over human interactions through efficiency, confidentiality, and reliability; noted usability deficits included the chatbot’s inability to personalize responses. In the domain of credibility, participants noted the institutional affiliation with Johns Hopkins as an asset and Vira’s inappropriately cartoonish visual persona as being an area for improvement. Young people and health workers, most of whom were already vaccinated, envisioned using the chatbot as a discussion aid to encourage others to seek out vaccines. Finally, interview participants offered clear guidance to comprehensively redesign Vira’s visual persona.

Vira’s usability scores compare favorably to those of other health chatbots evaluated through standardized measures. One study examining a health chatbot with majority White adults found a mean score of 61.6 out of 100, whereas an HPV vaccine counseling chatbot used by 24 mostly White young adults scored between 74 and 80 out of 100 [19,48]. Vira was very easy to use; provided useful, appropriate, and informative responses; and explained its purpose. Although only half of the respondents thought Vira understood well and coped with input errors, this is double the score seen in evaluations of other health chatbots, showing users’ expectations for response accuracy are high [49]. In interviews, participants appreciated that the chatbot—which can handle typos and shorthand (eg, “vax” for vaccine)—could understand full sentences, and they perceived a social-like encounter. In other chatbots, so-called social bonding increases user acceptability and confidence, influences persuasiveness, and alleviates anxiety [47,50]. Since users felt less self-conscious of how they phrased a question, they may have been freer to ask sensitive questions—or encourage others to do so from the privacy of their screens. The natural typing style gave the chatbot a human-like status, and the exchange became like a social interaction where there would be no real-world consequences for a perceived stupid or inappropriate question.

Users in our study noted that the chatbot was not personalized for them and did not customize responses regarding their baseline knowledge, attitudes, vaccination status, or individual health status (eg, underlying conditions). Many other health chatbots provide personalized content and conversations to improve user engagement, dialog quality, and self-reflection [51]. This is common across downloadable apps; a review of 78 health apps with chatbots noted that 60% of these included personalization features, with most (90%) apps personalizing content—some simply addressing users by name [18]. Vira’s design as a web-based app limited such functionality. However, participants cognizant of social discord around vaccination recognized the potential of the confidentiality and privacy offered by the anonymous web-based chatbot platform. In that the main challenge regarding personalization is privacy [37], this anonymity and the “safe space” offered by Vira may be weighed against personalization.

In terms of its credibility, Vira was rated as very friendly, with participants describing the tool as having the potential—with design iterations—to be a trustworthy source for information on COVID-19 vaccines, a politicized and emotive issue. Evidence supports the notion that chatbots presenting information about controversial topics can be convincing and trustworthy, especially with supporting links [52,53]. Many users of health chatbots report high satisfaction and positive perceptions, and the use of even moderately rated chatbots has been associated with behavior changes [19,20]. Although this study was not designed to evaluate the effectiveness of Vira in changing participant attitudes, their appraisal of the chatbot as being highly usable supports potential pathways toward behavior change to explore further.

Participants stated an interest in using Vira in their personal lives and, in the case of health workers, in professional roles. Large health organizations evidently understand the potential of chatbot technology, with the health care chatbot market expected to reach nearly $US 1 billion by 2027 [54]. Health agencies and some US states have also launched health chatbots during the COVID-19 pandemic to encourage widespread sharing of credible health information [55-58]. Further investments in high-performing, user-validated chatbots would aid health educators in communicating about rapidly changing vaccination guidance. In addition, installing chatbots on health department websites could reduce call volume and support public health workers [59].

This formative study provided investigators with user feedback to iteratively improve the user experience for Vira, a chatbot designed rapidly to support vaccine queries during the pandemic’s peak, including on the visual persona, provided rapid feedback for a website redesign (see Figure 1B). The new VIRA, spelled in all capital letters, is shown in calming blue and purple tones as a smaller, still-friendly orb supporting human users. The chatbot’s response database, or repository of potential responses, was also comprehensively edited via a separately reported message testing study [30].

However, numerous questions remain: What are the social implications of automating conversations about vaccine decisions, previously a person-to-person encounter highly reliant on trust? As mental health chatbots can reduce anxiety and loneliness, can vaccine chatbots simulate a support system validating people’s search for answers—helping them feel heard, even if by a bot? Evidence is needed, through a randomized evaluation, to explore which elements of a chatbot interaction, such as chat duration or added personalization, could lead to measurable changes in vaccine attitudes and behavior or, indeed, impact related to other stigmatized health issues such as sexual health. Whether chatbots effectively counter health misinformation and support the use of motivational interviewing, one of the only evidence-based means to soften vaccine hesitancy, is another important area for exploration [60,61].

Although this study provided rapid feedback to course-correct Vira’s visual design and inform its outreach strategy, it has several limitations. First, our participants were mostly college-educated, perhaps due to a reliance on Twitter ad recruitment [62]. In addition, participants were recruited using Johns Hopkins–branded ads and may have been more favorable toward the institution than others who did not click on the ads. IDI recruitment by Johns Hopkins of several health workers employed in Baltimore likely introduced bias, since participants may have been less likely to offer unfavorable comments out of a sense of collegiality; nonetheless, such participants did offer specific critiques. The conduct of the study at a university widely known to promote COVID-19 vaccination likely dissuaded some vaccine skeptics [63]. Since investigators needed to collect data rapidly to alter a tool already in use during a pandemic, we conducted a small number of qualitative interviews, focusing on professional users. Nearly all our participant sample said they had been vaccinated, which is not representative of young adults and health workers in the United States; however, a large proportion of the US public who got primary doses of COVID-19 vaccines have not subsequently obtained booster doses [64]. Therefore, we believe the results are relevant to support efforts to counteract vaccine hesitancy. Further, most participants describe theoretical usefulness in a one-time encounter with the chatbot. A final limitation is that this study describes Vira’s performance at launch; since then, the number of Key Points Vira can address has nearly doubled, and the performance of its adaptive algorithm has presumably improved.

Launched at the peak of the COVID-19 pandemic, as cases caused by the Delta variant crested in the US, Vira offered a highly functional and credible system to respond quickly and appropriately to users’ vaccine queries. We used rich data gathered through interviews to identify and remedy deficits in the chatbot persona. Young people and health workers in the study felt chatbots offered significant benefits in a pandemic context due to their reliability, responsiveness, and efficiency and that the Vira chatbot was a credible and private way to seek information on a sensitive issue. More research is needed to determine how guidance offered in an anonymous 2-way dialog, potentially designed to simulate a motivational interview, could shift perceptions of emotionally charged issues with participants in a real-world setting. Evidence is also needed to measure whether chatbots strengthen public education services and are cost-effective if made widely available.