A multistep, mixed methods, user-centered study was developed and run for 5 months. The goal of this research program was to measure the qualitative and quantitative experiential effects of an AI-based search and summarization tool (referred to as Expanse Navigator) within MEDITECH’s Expanse and to understand the feasibility of incorporating user-centered methods during the implementation phases of AI-based clinical tools.

Mile Bluff Medical Center is a 40-bed acute-care hospital and outpatient care center in Mauston, Wisconsin, United States. A total of 176 health care professionals were provided access to a clinical pilot of Expanse Navigator in January 2024; this includes 53 medical doctors or advanced practice providers, 61 registered nurses, 48 allied health professionals, and 14 health information management specialists across 24 distinct clinical and operational departments.

All pilot participants were invited to participate in the user-centered research program. A sample of 128 out of the 176 (72%) pilot program members opted into the program.

Expanse Navigator offers advanced search capabilities and an AI-generated, comprehensive view of a patient’s health history (Figures 2 and 3; Multimedia Appendix 1). It works together with and is accessible directly within MEDITECH’s Expanse. The tool offers new capabilities, unlocks valuable data (eg, PDFs and faxes), and makes them quickly and easily accessible to users via a modern search experience. The search capabilities handle misspellings and abbreviations, provide suggested search terms, and give results ranked by relevance. The tool is powered by AI rather than keyword matching.

The first component of the user-centered program was clinical observation. Before the deployment of Expanse Navigator, 21 representative users from the hospital, ambulatory care, outpatient clinics, and the emergency department were observed in their normal clinical environments (outside of patient care). Observations provided an understanding of users’ existing tools, workflows, environments, and experiences before the deployment of Expanse Navigator. Further, this effort helped to establish collaboration and build rapport with clinical users.

The second component of the user-centered program focused on experiential measurement through the use of pre- and postdeployment surveys. To develop the survey measures, MEDITECH and Google jointly developed a set of primary metrics: user satisfaction, product helpfulness across specified tasks (Multimedia Appendix 2), and perceived time savings. Survey questions used fully labeled 5-point scales, following industry best practices [27], and are available in the Multimedia Appendix 3. Survey questions were optional; respondents were not required to answer all questions.

Before deploying Expanse Navigator, we measured prelaunch metrics of MEDITECH’s Expanse via a baseline survey. Four weeks following deployment, we distributed a postlaunch survey to collect identical metrics on users’ experiences with Expanse Navigator.

Surveys were administered to all pilot program participants. There were 94 out of 176 (73%) responses and 73 out of 176 (57%) responses to the baseline survey and postlaunch survey, respectively.

The third component of the user-centered program involved early integration of product testing. The purpose of this assessment was to test the integrated product’s ability to support key clinical tasks, to collect feedback based on users’ first exposure to the product, and to ensure that no issues that could prevent easy and efficient use were present ahead of widespread launch.

During a single 60-minute session, participants viewed 3 minutes of prerecorded onboarding content and had 5 minutes of unstructured exploration with the product. Immediately thereafter, participants were prompted with 12 structured tasks to complete using Expanse Navigator features (Multimedia Appendix 4). Tasks were scored as “completed,” “completed with difficulty,” or “not completed.” The tasks were informed by findings from naturalistic observation sessions and designed to reflect responsibilities fulfilled in the course of daily work. Sessions were conducted until data confidently showed that users could complete key tasks. In our case, this research phase ended after sessions with 3 users showed high levels of success. This methodology facilitated unique feedback related to intuitiveness and ease of use and also highlighted areas of focus for user onboarding and education.

The final component of the user-centered program was qualitative interviews. Approximately 6 weeks after launching Expanse Navigator, we conducted 60-minute interviews with 4 users. We sought to capture qualitative data around the impact of the product, users’ experiences, high-priority use cases, and self-reported engagement. Participants spanned distinct roles and specialties and had not previously participated in one-on-one user-centered research sessions.

Completed survey responses were analyzed using descriptive statistics (eg, percentage of users reporting satisfaction, perceived helpfulness, and time savings). To measure the experiential impact of the AI tool, a within-subjects Mann-Whitney U test (1-tailed) was run on each metric, using data from respondents who completed both baseline and postlaunch surveys. Qualitative interviews were analyzed by conducting a thematic analysis of detailed interview notes [28].

This research was reviewed by the Advarra institutional review board and determined to be exempt from IRB oversight. All participants provided informed consent before participating.

The results of this study not only demonstrate that the AI-powered search and clinical discovery tool led to significant improvements in user satisfaction, helpfulness, and perception of time saved, but also demonstrate the feasibility of evaluating the clinical experience through a mixed method, user-centered approach.

AI-based tools are becoming increasingly common in clinical settings. As these technologies continue to advance, so will user expectations for high-quality product experiences. It will become increasingly important to measure the user experience of these tools with robust approaches that can be flexibly applied within a clinical environment. This research program was intentionally designed to evaluate product experience, refine understanding of user mental models, and optimize our onboarding program for the greatest impact when deploying an AI tool.

Incorporating user-centered research during predeployment stages allowed for the collection of benchmark measures of user satisfaction, helpfulness, and speed. It also informed the design of onboarding experiences and custom training for cohorts with specialized workflows. Furthermore, the activities during this stage led to a strong partnership with hospital leadership, which, in turn, increased awareness about the upcoming product launch.

The deployment and postdeployment research allowed the team to measure the impact of the AI tool from a clinical user’s point of view. The results of our case study demonstrate that the AI-based search and summarization experience, accessible directly within an EHR, provides users with new capabilities and increased levels of satisfaction, helpfulness, and speed.

Both the lift and feasibility of a user-centered approach were demonstrated through strong user participation, survey completion rates, and the overall duration of data collection (60 minutes per one-on-one study session).

The design of this study program follows previous work showing the importance of assessing the user experience during all phases of the product development life cycle: design, development, and deployment [29,30], and addresses calls for an increase in human-centered evaluative research alongside AI clinical deployments [31]. It extends past research that outlines approaches to user-centered research during design and development stages of AI-based clinical tools [19-21], and provides a framework for assessments that can occur during the later stages of clinical implementation.

There are several limitations to this research. First, while the research involved a large percentage of users for the AI-based tool within MEDITECH’s Expanse, across a wide range of clinical roles and specialities, it did so with users from a single medical center. Though we demonstrate feasibility for the user-centered study program presented in this paper, future research is needed to demonstrate generalizability. Second, while the findings from the case study show positive improvements in the user experience for this AI-based EHR integration, further research would be needed to demonstrate the impact across additional clinical measures.

By developing and executing a mixed methods, user-centered program for AI tool deployment, we enabled trust, collaboration, and frequent communication between clinical end users and AI practitioners, and a robust collection of key product metrics. This data-driven and user-centered approach championed end users’ experiences and surfaced those findings pre- and postproduct launch, highlighting specific areas for curated user training and optimizing the opportunity for rapid deployment. Integrating user-centered research as part of the deployment process ensures that AI tools are not only technologically advanced but are also relevant, usable, useful, and safe for use in any setting; notably, all of these benefits are of critical importance for tools used in health care and clinical settings.

Prioritizing a human-centered research program for AI-based tool deployment supports effective product integration and user satisfaction; further, it supports the thoughtful delivery of products and services in high-stakes health care environments where patient safety is the utmost priority.