Sutter Health is a large not-for-profit health care system serving more than 3 million people annually across 100 rural and urban communities in Northern California. Sutter Health was the first health care system in the United States to implement Epic System’s MyChart patient portal (My Health Online) in 2001 [27,28]. As of 2020, there have been over 3 million patients enrolled in My Health Online, which can be accessed via the website or through the mobile app, to communicate with their care team, refill prescriptions, view lab results, pay bills, and schedule appointments (including video visits).

In September 2017, we conducted pilot testing to assess the application and effectiveness of behavioral economics–driven language in encouraging patients to schedule appointments using the patient portal with the 2 largest of the 5 Sutter Health medical foundations (Palo Alto Medical Foundation and Sutter Medical Foundation) in the Sacramento Valley. There were 775,000 Palo Alto Medical Foundation patients enrolled in My Health Online with 222,000 unique log-ins in September 2017 and 41,000 appointments scheduled through the patient portal. There were 436,000 Sutter Medical Foundation patients enrolled in My Health Online, with 103,000 unique log-ins in September 2017 and 13,000 appointments scheduled through the patient portal. We selected 2 sites to demonstrate that nudge health maintenance reminder could be useful for more than one patient population. This also allowed us to understand potential confounders and variables before scaling across the organization.

The pilot consisted of testing of 2 message options, followed by the roll-out of the message for which the most patients scheduled appointments through the patient portal. We tested 2 themes—curiosity (email A) and exclusivity (email B)—in patient messaging (Figure 1). We selected these themes based on guidance from behavioral economics experts at VAL Health based upon their previous experiences [29-31]. Among 550,000 My Health Online members from Palo Alto Medical Foundation and Sutter Medical Foundation who had never scheduled appointments through the patient portal, 3800 patients were randomized to receive the promotional emails. More patients, of both Sutter Medical Foundation and Palo Alto Medical Foundation, scheduled appointments through the patient portal after receiving email B, thus email B was selected for use in the next phase.

Email B was sent to the remaining Palo Alto Medical Foundation and Sutter Medical Foundation members who had never scheduled appointments through the portal (n=550,000) on November 10, 2017. We included a control group of 10% of Palo Alto Medical Foundation and Sutter Medical Foundation My Health Online users to see if there was any difference in rates of appointments scheduled through the patient portal compared to those for individuals who received nudge emails.

The same behavioral economics concept was subsequently applied to health maintenance reminder messages in the patient portal. Nudge message wording was redesigned to use the exclusivity theme, with embedded functionality to click to schedule appointments through the patient portal for Medicare wellness visits and Pap smear (Figure 2). We selected Medicare wellness visits because it was already part of an active initiative at Sutter Health to increase utilization and we hoped that this type of message would better connect with patients who were eligible for Medicare wellness visits. We selected Pap smear so that we could examine how scheduling functions may impact screening completion and disease prevention. Nudge health maintenance reminders for Medicare wellness visits were implemented for all 5 Sutter medical foundations on November 15, 2017 (Figure 3). Nudge health maintenance reminders for Pap smear were launched on February 18, 2018; patients at Palo Alto Medical Foundation and Sutter Medical Foundation received nudge health maintenance reminders, while patients at the other 3 Sutter medical foundations continued to receive standard health maintenance reminders (control). Health maintenance reminders for Medicare wellness visits and Pap smear were discontinued during the COVID-19 pandemic; therefore, we analyzed utilization between January 2017 and February 2020.

The study was reviewed by the Sutter Health Institutional Review Board (SHIRB) and approved as a quality improvement study (IORG0004135).

Of 125,369 health maintenance reminders sent for Medicare wellness visits, 77% (96,839) were nudge health maintenance reminders (ie, using behavioral economics–based language). Of 125,369 Medicare wellness visits health maintenance reminder portal messages sent, 60.2% (75,407) led to an appointment being scheduled, with 12.2% (15,342) scheduled through the patient portal, and 51.5% (64,616) appointments completed. Of 585,358 health maintenance reminders sent for Pap smear, 49.1% (287,149) were nudge health maintenance reminders. Of 585,358 Pap smear health maintenance reminder portal messages sent, 21.9% (128,329) led to appointment being scheduled, 2.3% (13,259) scheduled through the patient portal, and 21.9% (128,255) appointments completed (Table 1).

Of 43,889 patients (age: mean 75 years, SD 7.2) included in the Medicare wellness visit analysis, 60.7% (26,662/43,889) were women. Approximately two-thirds (33,837/43,889, 77.1%) were White, and 41% (18,175/43,889) had no comorbid conditions. The mean number of encounters in the 12 months prior to the baseline date was 22 (SD 20.6). Of 288,152 women (age: mean 41 years, SD 12.8) included in the Pap smear analysis, there were diverse racial/ethnic groups (White: 141,426/288,152, 49.1%; Black: 8919/288,152, 3.1%; Hispanic: 37,398/288,152, 13.0%; Asian: 59,225/288,152, 20.6%; other: 41,184/288,152, 14.3%); 89% (255,796/288,152) had no comorbid conditions. The mean number of encounters in the 12 months prior to baseline was 7 (SD 12.0); 45.9% (132,151/288,152) had coverage through preferred provider organization or fee-for-service plans, and 14.4% (41,618/288,152) had coverage through health maintenance organization plans (Table 2).

For Medicare wellness visits, there was an increasing trend in proportion of appointments scheduled through the patient portal (Figure 4), and for Pap smear, appointments scheduled through the patient portal and appointment completion in the nudge group were consistently higher than those for the control group throughout the study period.

When comparing the unadjusted rates, we observed that a higher percentage of patients scheduled Medicare wellness visits through the patient portal after nudge implementation (nudge) than before implementation (control) for the overall sample (nudge: 12,573/96,839, 13.0%; control: 2769/28,530, 9.7%; P<.001) as well as for the subset with appointment scheduled (nudge: 12,573/58,371, 21.5%, control: 2769/17,036 16.3%; P<.001) (Table 3). A similar pattern for appointment scheduling through the patient portal was found for Pap smear between February 2018 and February 2020 for those who received health maintenance reminders (nudge: 8239/287,149, 2.9%, control: 1868/120,047, 1.6%; P<.001) and for the subset with appointment scheduled (nudge: 12.2%, control: 9.2%, P<.001), and the rate of appointment completion for Pap smear was higher (P<.001) in the nudge group (67,399/287,149, 23.5%) than that in the control group (20,393/120,047, 17.0%), while rates of appointment completion for Medicare wellness visits were comparable (nudge: 49,835/96,839, 51.5%; control: 14,781/28,530, 51.8%; P=.30).

Findings from segmented regression for Medicare wellness visits (Table 4) suggested that there was a marginal gradual effect of nudge messaging on Medicare wellness visits scheduled through the patient portal (monthly rate 0.26%; P=.09). There was a statistically significant increase, at a rate of 0.46% per month (P=.04), in appointments scheduled through the patient portal. There was no immediate effect for either patterns in appointments scheduled though the patient portal or for patterns in appointments completed for Medicare wellness visits.

The odds of scheduling a Pap smear for patients who received nudge health maintenance reminders were 1.62 times those for patients who received control health maintenance reminders (propensity adjusted odds ratio [OR] 1.62, 95% CI 1.50-1.74). A similar association was observed among the patients with scheduled appointments (propensity adjusted OR 1.61, 95% CI 1.47-1.76). Nudge health maintenance reminders were associated with a 7% increase in the odds of appointment completion for Pap smear (propensity adjusted OR 1.07, 95% CI 1.04-1.10).

We evaluated linear and categorical specifications (quintiles and deciles) of the propensity score. The categorical specification in deciles was selected, although sensitivity analyses using different specifications of the propensity score did not change our conclusions. Nudge health maintenance reminders were associated with a higher rate of scheduling through the patient portal for all health maintenance reminders and for patients who received health maintenance reminders and who scheduled appointments. For all health maintenance reminders, ORs ranged between 1.39-1.63 (quintile specification: adjusted OR 1.63, 95% CI 1.51-1.75; linear specification: adjusted OR 1.39, 95% CI 1.32-1.47). For patients with scheduled appointments, ORs ranged between 1.47-1.56 (quintile specification: adjusted OR 1.56, 95% CI 1.44-1.70; linear specification: adjusted OR 1.47, 95% CI 1.36-1.58). Nudge health maintenance reminders were associated with 10% to 17% increases in the odds of appointment completion for Pap smear (quintile specification: adjusted OR 1.17, 95% CI 1.14-1.20; linear specification: adjusted OR 1.10, 95% CI 1.07-1.12).

To the best of our knowledge, this is the first attempt to employ behavioral economics with electronic medical record portal–based health maintenance reminders to improve patient engagement and the utilization of preventive health services. Our findings suggest that our intervention influenced patient behavior. Simple modifications to verbiage and the framing of messages and adding embedded scheduling functionality had an impact on patients’ actions. Although we did not find an immediate effect of nudge on scheduling appointments through the patient portal, we observed a sustained effect over time especially among those who scheduled an appointment. We believe this is because once patients learned how to schedule appointments through the patient portal, they used the portal to schedule appointments from that time onward. Similarly, we found that nudge health maintenance reminders for Pap smear were associated with a 65% to 68% increase in the odds of scheduling appointments through the patient portal, compared to control health maintenance reminders during the same study timeframe.

Appointments have typically been scheduled over the phone or in person, which can be resource intensive. Patient portals with web-based scheduling improves patient convenience, flexibility, and communications with providers, while reducing administrative burdens [34,35]. Furthermore, the COVID-19 pandemic has increased preference for and use of telehealth and features such as web-based scheduling. Behavioral economics–based health maintenance reminders are low-cost, effective, and operationally feasible. Once designed and pilot tested, they can be centrally deployed to all eligible patients with relatively low administrative burden and costs.

We acknowledge the very low rate for scheduling appointments through the patient portal for Pap smear (13,259/585,358, 2.3%) and posit that this may be due to a couple of factors. First, at Sutter Health, the majority of obstetric and gynecological practices do not allow appointments to be scheduled through the patient portal. Patients could schedule appointments with their primary care provider through the portal but could only type “Pap smear” in the free text box if they would like to receive one during that visit. Since there is no Pap smear visit scheduling type in the electronic medical record system, it added some complexity in retrieving these data—we identified the Pap smear procedure in subsequent visits. Nevertheless, even a small percentage means that more patients than those who would have with a standard message were able to receive this preventative screening.

Proactively engaging patients to schedule their appointments is the important first step to care management. Follow-through appointment completion is next. Our findings are mixed when examining the effect of nudge on appointment completion. We did not observe a significant effect on appointment completion for Medicare wellness visits (P=.30). The nudge effect for Pap smear was moderate, with a 7% increase, compared to control health maintenance reminders. The difference in nudge effects between Medicare wellness visits and Pap smear may be attributable to differences in patient populations to some degree. Exploratory stratified analyses suggested that patients who were younger (vs those 65 years and older) and Asian (vs other race or ethnicity groups) had the highest Pap smear appointment completion rate and most improvement from receiving nudge messages. Compared to the Medicare wellness visit sample, the Pap smear sample represented a younger patient population, with relatively more Asian individuals. We also explored if the number of health maintenance reminders sent plays a role in appointment completion. Among those who received multiple health maintenance reminders, subsequent health maintenance reminders were associated with higher rates of appointment completion than those for the first health maintenance reminders. These patterns were similar for both Medicare wellness visits and Pap smear. Scheduling an appointment or procedure does not automatically lead to completion, and thus, completion is a more complex process than scheduling and likely to be influenced by additional factors, which requires further investigation.

Our study has several limitations. First, our analyses are based on data from a large health care system in Northern California. Our nudge intervention was designed for patients with access to internet and email which limits the generalizability of specific estimates of the nudge effect. Second, the propensity score covariate–adjusted method was selected because it mitigates confounding in observational studies such as ours. We chose this method over other widely used propensity score–matching methods. Due to the case-control ratio in our data (approximately 2.4:1 between February 2018 and February 2020), the propensity score–matching approach would have resulted in a loss of statistical power. The rollout of nudge health maintenance reminders was designed with operational goals to encourage patient uptake of web-based tools and to facilitate scheduling, which is expected to lead to better care management and health outcomes. As such, more cases than controls were enrolled. Different specifications of propensity score modeling may affect the results of our propensity score covariate–adjusted approach. We conducted sensitivity analyses using linear and nonlinear specifications of propensity score and our findings were consistent. Third, we focused on the overall patterns. Future research is needed to understand potential variability by specific subgroups (race/ethnicity, socioeconomic status) to inform targeted, culturally appropriate designs to maximize the benefit of nudges. The time of the year may have played a role in appointment scheduling and completion. Our study period limited the ability to examine the seasonal effect. We also recognize that the generalizability of these findings might be limited, with respect to application to other patient portals, as many have different functionalities and user experiences than those in the portal used in this study.

Nudges, a behavioral economics–based health maintenance reminder, improve web-based scheduling and subsequent appointment completion for Medicare wellness visits and Pap smear, with important long-term impacts. Given these results, Sutter Health implemented messages with behavioral economics–based language for all other health maintenance reminders on May 28, 2020. Future studies should explore why nudge worked for some patients and not others.