This paper is in the following e-collection/theme issue:

Design of Processes and Workflows (102) Health Services Research (482) Prevention and Health Promotion (986) Personal Health Records, Patient-Accessible Electronic Health Records, Patient Portals (573) Organizational Issues (154) Adoption and Change Management of eHealth Systems (712) Obstetrics, Gynecology, and Reproductive Medicine (225)

Published on in Vol 9, No 1 (2022): Jan-Mar

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/34090, first published .
Effects of Behavioral Economics–Based Messaging on Appointment Scheduling Through Patient Portals and Appointment Completion: Observational Study

Effects of Behavioral Economics–Based Messaging on Appointment Scheduling Through Patient Portals and Appointment Completion: Observational Study

Effects of Behavioral Economics–Based Messaging on Appointment Scheduling Through Patient Portals and Appointment Completion: Observational Study

Authors of this article:

Su-Ying Liang1 Author Orcid Image ;   Cheryl D Stults1 Author Orcid Image ;   Veena G Jones2 Author Orcid Image ;   Qiwen Huang1 Author Orcid Image ;   Jeremy Sutton2 Author Orcid Image ;   Guy Tennyson2 Author Orcid Image ;   Albert S Chan2, 3 Author Orcid Image
Article Authors Cited by (4) Tweetations (3) Metrics

    Original Paper

    1Sutter Health Center for Health Systems Research, Palo Alto, CA, United States

    2Clinical Leadership Team, Sutter Health, Sacramento, CA, United States

    3Center for Biomedical Informatics Research, Stanford University, Stanford, CA, United States

    Corresponding Author:

    Su-Ying Liang, PhD

    Sutter Health Center for Health Systems Research

    795 El Camino Real, Ames Building

    Palo Alto, CA,

    United States

    Phone: 1 650 691 6432

    Email: liangs2@sutterhealth.org


    Background: Behavioral economics–based techniques have been an increasingly utilized method in health care to influence behavior change by modifying language in patient communication (through choice architecture and the framing of words). Patient portals are a key tool for facilitating patient engagement in their health, and interventions deployed via patient portals have been effective in improving utilization of preventive health services.

    Objective: We examined the impacts of behavioral economics–based nudge health maintenance reminders on appointment scheduling through a patient portal and appointment completion for 2 preventive services: Medicare wellness visits and Pap smear.

    Methods: We conducted a retrospective observational study using electronic health record data from an integrated health care system in Northern California. Nudge health maintenance reminders with behavioral economics–based language were implemented for all sites in November 2017 for Medicare wellness visits and for selected sites in February 2018 for Pap smears. We analyzed 125,369 health maintenance reminders for Medicare wellness visits, and 585,358 health maintenance reminders for Pap smear sent between January 2017 and February 2020. The primary outcomes were rate of appointments scheduled through the patient portal and appointment completion rate. We compared the outcomes between those who received the new, behavioral economics–based health maintenance reminders (the nudge group) and those who received the original, standard health maintenance reminders (the control group). We used segmented regression with interrupted time series to assess the immediate and gradual effect of the nudge for Medicare wellness visits, and we used logistic regression to assess the association of nudge health maintenance reminders, adjusting for the propensity to receive a nudge health maintenance reminder, for Pap smear.

    Results: The rates of appointments scheduled through the patient portal were higher for nudge health maintenance reminder recipients than those for control health maintenance reminder recipients (Medicare wellness visits—nudge: 12,537/96,839, 13.0%; control: 2,769/28,530, 9.7%, P<.001; Pap smear—nudge: 8,239/287,149, 2.9%; control: 1,868/120,047, 1.6%; P<.001). Rates of appointment completion were higher for nudge health maintenance reminders for Pap smear (nudge: 67,399/287,149, 23.5% control: 20,393/120,047, 17.0%; P<.001) but were comparable for Medicare wellness visits (nudge: 49,835/96,839, 51.5% control: 14,781/28,530, 51.8%; P=.30). There was a marginally gradual effect of nudge on number of appointments scheduled through the patient portal for the overall Medicare wellness visits sample (at a monthly rate of 0.26%, P=.09), and a significant gradual effect among scheduled appointments (at a monthly rate of 0.46%, P=.04). For Pap smear, nudge health maintenance reminders were positively associated with number of appointments scheduled through the patient portal (overall sample: propensity adjusted odds ratio [OR] 1.62; 95% CI 1.50-1.74; among scheduled appointments: propensity adjusted OR 1.61, 95% CI 1.47-1.76) and with appointment completion (propensity adjusted OR 1.07; 1.04-1.10).

    Conclusions: Nudges, a behavioral economics–based approach to providing health maintenance reminders, increased the number of appointments scheduled through the patient portal for Medicare wellness visits and Pap smear. Our study demonstrates that a simple approach—framing and modifying language in an electronic message—can have a significant and long-term impact on patient engagement and access to care.

    JMIR Hum Factors 2022;9(1):e34090

    doi:10.2196/34090

    Keywords

    access to care; behavioral economics; online; web-based appointment scheduling; health service; behavior; health care 


    Health care has become more directly accessible to the patient, in part due to the Health Information Technology for Economic and Clinical Health Act of 2009 [1], and also, as a result of increasing consumer demands [2].When first introduced, patient portals provided patients with limited access to their medical records and adoption was low [3]. In part to meet meaningful use requirements of the Health Information Technology for Economic and Clinical Health Act, web-based electronic health record portal capabilities have expanded to appointment scheduling, displaying lab results or viewing encounter notes written by their physician [4], allowing bills payments, and facilitating communication with care teams, costs estimates for ambulatory services [5], and access to family records. Patient access rates to web-based patient portals have increased to 90% in some organizations [6]. Yet, access alone can only go so far in engaging patients with patient portals. Message construction and delivery also influence patient utilization of patient portals [7].

    In the field of behavioral economics, researchers recognize that humans do not always act logically [8] in making choices and provide tools to help influence desired behaviors. One strategy includes choice architecture interventions, in which the presentation of options are altered to improve decision-making without restricting choice [9]. Subtle design changes, such as reordering choices, limiting options, or modifying a default can significantly influence behavior. When applying these tools and others, choice architects operate with a key tenet in mind: reduce friction in decision-making to make the desired path the one of least resistance [8,10]. In addition to the choice environment, behavioral economists also know that the framing of words can nudge individuals in a given direction of action [9]. For instance, messages that use common language [11], harness aversion to loss and regret [11,12], or embed social elements and devices [11-13] into their core are likely to be very effective at driving behavior change [11-13].

    Over the past decade, these tenets of behavioral economics have been increasingly utilized in health care for both patients (cancer screenings [14], hospital appointment no-show rates [7], medication adherence for behavioral health [15], HIV testing rates [16], obesity and binge eating [17]) and clinicians (prescribing behavior [18,19], overtreatment of diabetes [20]), with 83 publications in the top 3 highest impact general medicine journals (Journal of the American Medical Association, The Lancet, The New England Journal of Medicine) from 1998 to 2018 [21]. A systematic review [22] found patient portal interventions to be effective in improving a few psychological outcomes, medication adherence, and preventive service use. Yet, to the best of our knowledge, behavioral economics has not been applied to health maintenance reminders sent through web-based patient portals to improve patient engagement and increase preventive service use for annual Medicare wellness visits and Pap smear. General use of the Medicare wellness visits has increased over time [23], with 7% [24] to 8.1% [25] of Medicare beneficiaries receiving an annual wellness visit in 2011 increasing to 16% in 2014 [24] and 23% in 2016 [25]. Although rates for Pap smear are much higher, with 83.7% of women age 21 to 65 years reporting having one within the past 3 years in 2018 [26], there is still room for improvement. Our objective was to determine if small changes in the wording of health maintenance reminders could alter patient completion of these preventive health services.


    Setting

    Sutter Health is a large not-for-profit health care system serving more than 3 million people annually across 100 rural and urban communities in Northern California. Sutter Health was the first health care system in the United States to implement Epic System’s MyChart patient portal (My Health Online) in 2001 [27,28]. As of 2020, there have been over 3 million patients enrolled in My Health Online, which can be accessed via the website or through the mobile app, to communicate with their care team, refill prescriptions, view lab results, pay bills, and schedule appointments (including video visits).

    Pilot Testing: Behavioral Economics–Based Email Messaging

    In September 2017, we conducted pilot testing to assess the application and effectiveness of behavioral economics–driven language in encouraging patients to schedule appointments using the patient portal with the 2 largest of the 5 Sutter Health medical foundations (Palo Alto Medical Foundation and Sutter Medical Foundation) in the Sacramento Valley. There were 775,000 Palo Alto Medical Foundation patients enrolled in My Health Online with 222,000 unique log-ins in September 2017 and 41,000 appointments scheduled through the patient portal. There were 436,000 Sutter Medical Foundation patients enrolled in My Health Online, with 103,000 unique log-ins in September 2017 and 13,000 appointments scheduled through the patient portal. We selected 2 sites to demonstrate that nudge health maintenance reminder could be useful for more than one patient population. This also allowed us to understand potential confounders and variables before scaling across the organization.

    The pilot consisted of testing of 2 message options, followed by the roll-out of the message for which the most patients scheduled appointments through the patient portal. We tested 2 themes—curiosity (email A) and exclusivity (email B)—in patient messaging (Figure 1). We selected these themes based on guidance from behavioral economics experts at VAL Health based upon their previous experiences [29-31]. Among 550,000 My Health Online members from Palo Alto Medical Foundation and Sutter Medical Foundation who had never scheduled appointments through the patient portal, 3800 patients were randomized to receive the promotional emails. More patients, of both Sutter Medical Foundation and Palo Alto Medical Foundation, scheduled appointments through the patient portal after receiving email B, thus email B was selected for use in the next phase.

    Email B was sent to the remaining Palo Alto Medical Foundation and Sutter Medical Foundation members who had never scheduled appointments through the portal (n=550,000) on November 10, 2017. We included a control group of 10% of Palo Alto Medical Foundation and Sutter Medical Foundation My Health Online users to see if there was any difference in rates of appointments scheduled through the patient portal compared to those for individuals who received nudge emails.

    ‎
    Figure 1. Themes of pilot behavioral economics–based email messages.
    View this figure

    Application to Health Maintenance Reminders

    The same behavioral economics concept was subsequently applied to health maintenance reminder messages in the patient portal. Nudge message wording was redesigned to use the exclusivity theme, with embedded functionality to click to schedule appointments through the patient portal for Medicare wellness visits and Pap smear (Figure 2). We selected Medicare wellness visits because it was already part of an active initiative at Sutter Health to increase utilization and we hoped that this type of message would better connect with patients who were eligible for Medicare wellness visits. We selected Pap smear so that we could examine how scheduling functions may impact screening completion and disease prevention. Nudge health maintenance reminders for Medicare wellness visits were implemented for all 5 Sutter medical foundations on November 15, 2017 (Figure 3). Nudge health maintenance reminders for Pap smear were launched on February 18, 2018; patients at Palo Alto Medical Foundation and Sutter Medical Foundation received nudge health maintenance reminders, while patients at the other 3 Sutter medical foundations continued to receive standard health maintenance reminders (control). Health maintenance reminders for Medicare wellness visits and Pap smear were discontinued during the COVID-19 pandemic; therefore, we analyzed utilization between January 2017 and February 2020.

    ‎
    Figure 2. Standard (control) and behavioral economics–based (nudge) health maintenance reminders.
    View this figure
    ‎
    Figure 3. Study design. HMR: health maintenance reminder; MWV: Medicare wellness visit.
    View this figure

    Statistical Analysis

    Study Sample

    For this retrospective, observational study with electronic health record data, we identified 125,369 Medicare wellness visit health maintenance reminders sent to 43,889 unique patients who were 65 years and older between January 1, 2017 and February 28, 2020, and we identified 585,358 health maintenance reminders for Pap smear sent to 288,152 unique patients who were 21 years and older during the same study period.

    Measures and Statistical Approach

    The primary outcomes were rates of appointments scheduled through the patient portal and completed. The predictor of interest in all analyses was the receipt of nudge health maintenance reminder. The 2-tailed t test was used to examine differences in the unadjusted proportions of appointments scheduled through the patient portal and appointment completion between the nudge and control groups. We also conducted a subgroup analysis focusing on patterns of proportion of appointments scheduled through the patient portal among scheduled appointments.

    For Medicare wellness visits, we used segmented regression with interrupted time series analysis to assess the immediate and gradual impact of the nudge intervention on study outcomes. The advantage of this analytic approach is its ability to distinguish the intervention effect from the secular trend (ie, a trend change that would have happened even in the absence of the intervention). The unit of analysis was the month. Study outcomes were measured in a given month: percentage of appointments scheduled through the patient portal (number of health maintenance reminders leading to appointment scheduled online / total number of health maintenance reminders) and percentage of appointment completion (number of health maintenance reminders with appointment completed / total number of health maintenance reminders).

    For Pap smear, after initial launch of nudge health maintenance reminders, there remained a mix of nudge and control health maintenance reminders between February 2018 and February 2020. We used logistic regression models to assess the association of nudge health maintenance reminders with study outcomes, adjusting for the propensity to receive a nudge health maintenance reminder and accounting for clustering within the patients. The unit of analysis was the health maintenance reminder. The outcomes were use of the patient portal to schedule an appointment (yes or no) and completion of the appointment (yes or no). The propensity score method was chosen to control for potential selection bias and confounding by factors associated with the intervention and the study outcomes [32,33]. We estimated the propensity to receive a nudge health maintenance reminder as a function of patient demographic characteristics (age and race/ethnicity: White, Black, Hispanic, Asian, other), Charlson comorbidity score at baseline, health care utilization (number of health care encounters including in-person visit, video visits, My Health Online messages, and telephone calls) in the 12 months prior to baseline, insurance type (preferred provider organization or fee-for-service, health maintenance organization, Medicare, Medicaid, other), and their primary care physician’s service location. Baseline was defined as the date of the first health maintenance reminder that the patient received during the analysis timeframe. Linear and categorical specifications of the propensity score were evaluated to ensure the robustness of the results. Analyses were conducted in R (version 4.0.4; The R Project) and Stata (version 16.1; StataCorp LLC).

    Ethics

    The study was reviewed by the Sutter Health Institutional Review Board (SHIRB) and approved as a quality improvement study (IORG0004135).


    Sample Characteristics

    Of 125,369 health maintenance reminders sent for Medicare wellness visits, 77% (96,839) were nudge health maintenance reminders (ie, using behavioral economics–based language). Of 125,369 Medicare wellness visits health maintenance reminder portal messages sent, 60.2% (75,407) led to an appointment being scheduled, with 12.2% (15,342) scheduled through the patient portal, and 51.5% (64,616) appointments completed. Of 585,358 health maintenance reminders sent for Pap smear, 49.1% (287,149) were nudge health maintenance reminders. Of 585,358 Pap smear health maintenance reminder portal messages sent, 21.9% (128,329) led to appointment being scheduled, 2.3% (13,259) scheduled through the patient portal, and 21.9% (128,255) appointments completed (Table 1).

    Of 43,889 patients (age: mean 75 years, SD 7.2) included in the Medicare wellness visit analysis, 60.7% (26,662/43,889) were women. Approximately two-thirds (33,837/43,889, 77.1%) were White, and 41% (18,175/43,889) had no comorbid conditions. The mean number of encounters in the 12 months prior to the baseline date was 22 (SD 20.6). Of 288,152 women (age: mean 41 years, SD 12.8) included in the Pap smear analysis, there were diverse racial/ethnic groups (White: 141,426/288,152, 49.1%; Black: 8919/288,152, 3.1%; Hispanic: 37,398/288,152, 13.0%; Asian: 59,225/288,152, 20.6%; other: 41,184/288,152, 14.3%); 89% (255,796/288,152) had no comorbid conditions. The mean number of encounters in the 12 months prior to baseline was 7 (SD 12.0); 45.9% (132,151/288,152) had coverage through preferred provider organization or fee-for-service plans, and 14.4% (41,618/288,152) had coverage through health maintenance organization plans (Table 2).

    Table 1. Health maintenance reminders sent and outcomes.
    Health maintenance reminderMedicare wellness visits (n=125,369), n (%)Pap smear (n=585,358), n (%)
    Type


    		Nudge96,839 (77.2)287,149 (49.1)

    		Control28,530 (22.8)298,209 (50.9)
    Appointment scheduled


    		No49,962 (39.8)457,029 (78.1)

    		Yes75,407 (60.2)128,329 (21.9)


    		Scheduled through patient portal




    		Yes15,342 (12.2)13,259 (2.3)



    		No110,027 (87.8)572,099 (97.7)


    		Appointment completed




    		Yes64,616 (51.5)128,255 (21.9)



    		No60,753 (48.5)457,103 (78.1)
    Table 2. Sample characteristics.
    CharacteristicMedicare wellness visits (n=43,889), n (%)Pap smear (n=288,152), n (%)
    Age (years), mean (SD)75.3 (7.2)40.9 (12.8)
    Gender, n (%)


    		Male17,267 (39.3)0 (0)

    		Female (%)26,622 (60.7)288,152 (100)
    Race/ethnicity, n (%)


    		White33,837 (77.1)141,426 (49.1)

    		Black1121 (2.6)8919 (3.1)

    		Hispanic2947 (6.7)37,398 (13.0)

    		Asian3476 (7.9)59,225 (20.6)

    		American Indian or Alaska Native/Pacific Islander or Native Hawaiian198 (0.4)1639 (0.6)

    		Multirace347 (0.8)5819 (2.0)

    		Unknown1963 (4.5)33,726 (11.7)
    Comorbidity score at baseline, n (%)


    		018,175 (41.4)255,796 (88.8)

    		18462 (19.3)22,167 (7.7)

    		27027 (16.0)6882 (2.4)

    		3+10,225 (23.3)3307 (1.1)
    Health care utilization at baseline, n (%)


    		Encountersa22.3 (20.6)6.6 (12.0)

    		Insurance type, n (%)



    		Health maintenance organization373 (0.9)41,618 (14.4)


    		Medicaid or Medi-Cal36 (0.1)9713 (3.4)


    		Medicare fee-for-service28,291 (64.4)7626 (2.7)


    		Medicare health maintenance organization14,168 (32.3)1045 (0.4)


    		Preferred provider organization or fee-for-service750 (1.7)132,151 (45.9)


    		Other/Unknown271 (0.6)95,999 (33.2)

    		Region (primary care physician’s location at baseline), n (%)



    		Region A12,864 (29.3)143,518 (49.8)


    		Region B4133 (9.4)19,969 (6.9)


    		Region C24,402 (55.6)62,988 (21.9)


    		Region D1681 (3.8)18,587 (6.5)


    		Region E763 (1.7)16,966 (5.8)


    		Other or no primary care physician46 (0.1) 26,135 (9.1)

    aEncounters included in-person visits, video visits, My Health Online message, and telephone.

    Appointment Scheduling and Completion

    For Medicare wellness visits, there was an increasing trend in proportion of appointments scheduled through the patient portal (Figure 4), and for Pap smear, appointments scheduled through the patient portal and appointment completion in the nudge group were consistently higher than those for the control group throughout the study period.

    When comparing the unadjusted rates, we observed that a higher percentage of patients scheduled Medicare wellness visits through the patient portal after nudge implementation (nudge) than before implementation (control) for the overall sample (nudge: 12,573/96,839, 13.0%; control: 2769/28,530, 9.7%; P<.001) as well as for the subset with appointment scheduled (nudge: 12,573/58,371, 21.5%, control: 2769/17,036 16.3%; P<.001) (Table 3). A similar pattern for appointment scheduling through the patient portal was found for Pap smear between February 2018 and February 2020 for those who received health maintenance reminders (nudge: 8239/287,149, 2.9%, control: 1868/120,047, 1.6%; P<.001) and for the subset with appointment scheduled (nudge: 12.2%, control: 9.2%, P<.001), and the rate of appointment completion for Pap smear was higher (P<.001) in the nudge group (67,399/287,149, 23.5%) than that in the control group (20,393/120,047, 17.0%), while rates of appointment completion for Medicare wellness visits were comparable (nudge: 49,835/96,839, 51.5%; control: 14,781/28,530, 51.8%; P=.30).

    ‎
    Figure 4. Trends of web-based scheduling and appointment completion from January 2017 to February 2020.
    View this figure
    Table 3. Rates of appointment scheduling through the patient portal and appointment completion between nudge and control groups.

    		Medicare wellness visitsPap smear




    		Control (January 2017-November 2017)Nudge (December 2017-February 2020)P valueControl (February 2018-February 2020)Nudge (February 2018-February 2020)P value
    Health maintenance reminders, n28,53096,839—a120,047287,149—

    		Appointments scheduled, n17,03658,371—20,40267,445—


    		Appointment scheduled through the patient portal, n276912,573—18688239—



    		Unadjustedb, %9.713.0<.0011.62.9<.001



    		Adjustedc, %16.321.5<.0019.212.2<.001

    		Appointments completed, n (%)14,781 (51.8)49,835 (51.8).3020,393 (17.0)67,399 (23.5) <.001

    aThe comparison was not made.

    bThe percentage was calculated using the number of health maintenance reminders.

    cThe percentage was calculated using the number of appointments scheduled.

    Nudge Effect

    Findings from segmented regression for Medicare wellness visits (Table 4) suggested that there was a marginal gradual effect of nudge messaging on Medicare wellness visits scheduled through the patient portal (monthly rate 0.26%; P=.09). There was a statistically significant increase, at a rate of 0.46% per month (P=.04), in appointments scheduled through the patient portal. There was no immediate effect for either patterns in appointments scheduled though the patient portal or for patterns in appointments completed for Medicare wellness visits.

    The odds of scheduling a Pap smear for patients who received nudge health maintenance reminders were 1.62 times those for patients who received control health maintenance reminders (propensity adjusted odds ratio [OR] 1.62, 95% CI 1.50-1.74). A similar association was observed among the patients with scheduled appointments (propensity adjusted OR 1.61, 95% CI 1.47-1.76). Nudge health maintenance reminders were associated with a 7% increase in the odds of appointment completion for Pap smear (propensity adjusted OR 1.07, 95% CI 1.04-1.10).

    We evaluated linear and categorical specifications (quintiles and deciles) of the propensity score. The categorical specification in deciles was selected, although sensitivity analyses using different specifications of the propensity score did not change our conclusions. Nudge health maintenance reminders were associated with a higher rate of scheduling through the patient portal for all health maintenance reminders and for patients who received health maintenance reminders and who scheduled appointments. For all health maintenance reminders, ORs ranged between 1.39-1.63 (quintile specification: adjusted OR 1.63, 95% CI 1.51-1.75; linear specification: adjusted OR 1.39, 95% CI 1.32-1.47). For patients with scheduled appointments, ORs ranged between 1.47-1.56 (quintile specification: adjusted OR 1.56, 95% CI 1.44-1.70; linear specification: adjusted OR 1.47, 95% CI 1.36-1.58). Nudge health maintenance reminders were associated with 10% to 17% increases in the odds of appointment completion for Pap smear (quintile specification: adjusted OR 1.17, 95% CI 1.14-1.20; linear specification: adjusted OR 1.10, 95% CI 1.07-1.12).

    Table 4. Regression analysis results.

    		Medicare wellness visits, coefficient (SE)Pap smeara, odds ratio (95% CI)

    		Preintervention level (Baseline)Preintervention slope (secular trend, per month)Change in intercept (immediate effect)Change in slope (gradual effect, per month)UnadjustedPropensity
    adjusted
    Scheduled through the patient portal9.421 (0.941)0.001 (0.151)−0.700 (0.971)0.264 (0.155)1.87 (1.78 to 1.97)1.62 (1.50 to 1.74)
    Scheduled through the patient portal among all scheduled16.343 (1.305)−0.043 (0.210)−1.071 (1.345)0.462 (0.215)1.38 (1.31 to 1.46)1.61 (1.47 to 1.76)
    Appointment completed50.687 (1.728)−0.007 (0.279)−0.201 (1.782)0.052 (0.285)1.50 (1.47 to 1.53)1.07 (1.04 to 1.10)

    aThe reference group in the model is patients who received a standard (control) health maintenance reminder. The propensity to receive a nudge health maintenance reminder was estimated as a function of patient age at baseline, race/ethnicity, Charlson comorbidity score at baseline, number of encounters at baseline, insurance type, and service location and was categorized in deciles in the adjusted model.


    Principal Findings

    To the best of our knowledge, this is the first attempt to employ behavioral economics with electronic medical record portal–based health maintenance reminders to improve patient engagement and the utilization of preventive health services. Our findings suggest that our intervention influenced patient behavior. Simple modifications to verbiage and the framing of messages and adding embedded scheduling functionality had an impact on patients’ actions. Although we did not find an immediate effect of nudge on scheduling appointments through the patient portal, we observed a sustained effect over time especially among those who scheduled an appointment. We believe this is because once patients learned how to schedule appointments through the patient portal, they used the portal to schedule appointments from that time onward. Similarly, we found that nudge health maintenance reminders for Pap smear were associated with a 65% to 68% increase in the odds of scheduling appointments through the patient portal, compared to control health maintenance reminders during the same study timeframe.

    Appointments have typically been scheduled over the phone or in person, which can be resource intensive. Patient portals with web-based scheduling improves patient convenience, flexibility, and communications with providers, while reducing administrative burdens [34,35]. Furthermore, the COVID-19 pandemic has increased preference for and use of telehealth and features such as web-based scheduling. Behavioral economics–based health maintenance reminders are low-cost, effective, and operationally feasible. Once designed and pilot tested, they can be centrally deployed to all eligible patients with relatively low administrative burden and costs.

    We acknowledge the very low rate for scheduling appointments through the patient portal for Pap smear (13,259/585,358, 2.3%) and posit that this may be due to a couple of factors. First, at Sutter Health, the majority of obstetric and gynecological practices do not allow appointments to be scheduled through the patient portal. Patients could schedule appointments with their primary care provider through the portal but could only type “Pap smear” in the free text box if they would like to receive one during that visit. Since there is no Pap smear visit scheduling type in the electronic medical record system, it added some complexity in retrieving these data—we identified the Pap smear procedure in subsequent visits. Nevertheless, even a small percentage means that more patients than those who would have with a standard message were able to receive this preventative screening.

    Proactively engaging patients to schedule their appointments is the important first step to care management. Follow-through appointment completion is next. Our findings are mixed when examining the effect of nudge on appointment completion. We did not observe a significant effect on appointment completion for Medicare wellness visits (P=.30). The nudge effect for Pap smear was moderate, with a 7% increase, compared to control health maintenance reminders. The difference in nudge effects between Medicare wellness visits and Pap smear may be attributable to differences in patient populations to some degree. Exploratory stratified analyses suggested that patients who were younger (vs those 65 years and older) and Asian (vs other race or ethnicity groups) had the highest Pap smear appointment completion rate and most improvement from receiving nudge messages. Compared to the Medicare wellness visit sample, the Pap smear sample represented a younger patient population, with relatively more Asian individuals. We also explored if the number of health maintenance reminders sent plays a role in appointment completion. Among those who received multiple health maintenance reminders, subsequent health maintenance reminders were associated with higher rates of appointment completion than those for the first health maintenance reminders. These patterns were similar for both Medicare wellness visits and Pap smear. Scheduling an appointment or procedure does not automatically lead to completion, and thus, completion is a more complex process than scheduling and likely to be influenced by additional factors, which requires further investigation.

    Limitations

    Our study has several limitations. First, our analyses are based on data from a large health care system in Northern California. Our nudge intervention was designed for patients with access to internet and email which limits the generalizability of specific estimates of the nudge effect. Second, the propensity score covariate–adjusted method was selected because it mitigates confounding in observational studies such as ours. We chose this method over other widely used propensity score–matching methods. Due to the case-control ratio in our data (approximately 2.4:1 between February 2018 and February 2020), the propensity score–matching approach would have resulted in a loss of statistical power. The rollout of nudge health maintenance reminders was designed with operational goals to encourage patient uptake of web-based tools and to facilitate scheduling, which is expected to lead to better care management and health outcomes. As such, more cases than controls were enrolled. Different specifications of propensity score modeling may affect the results of our propensity score covariate–adjusted approach. We conducted sensitivity analyses using linear and nonlinear specifications of propensity score and our findings were consistent. Third, we focused on the overall patterns. Future research is needed to understand potential variability by specific subgroups (race/ethnicity, socioeconomic status) to inform targeted, culturally appropriate designs to maximize the benefit of nudges. The time of the year may have played a role in appointment scheduling and completion. Our study period limited the ability to examine the seasonal effect. We also recognize that the generalizability of these findings might be limited, with respect to application to other patient portals, as many have different functionalities and user experiences than those in the portal used in this study.

    Conclusions

    Nudges, a behavioral economics–based health maintenance reminder, improve web-based scheduling and subsequent appointment completion for Medicare wellness visits and Pap smear, with important long-term impacts. Given these results, Sutter Health implemented messages with behavioral economics–based language for all other health maintenance reminders on May 28, 2020. Future studies should explore why nudge worked for some patients and not others.

    Acknowledgments

    We are grateful to the contributions made by VAL Health, in study design, and to James Read and Kevin Hays for their assistance and expertise with electronic health record data extraction for this study.

    Conflicts of Interest

    None declared.

    1. Health information technology for economic and clinical health (HITECH) act. US Department of Health and Human Services, and Ways and Means Committee. 2009.   URL: https:/​/www.​hhs.gov/​sites/​default/​files/​ocr/​privacy/​hipaa/​understanding/​coveredentities/​hitechact.​pdf [accessed 2022-03-10]
    2. Patel V, Barker W, Siminerio E. rends in consumer access and use of electronic health information. Office of the National Coordinator for Health Information Technology. 2015.   URL: https://www.healthit.gov/sites/default/files/briefs/oncdatabrief30_accesstrends_.pdf [accessed 2022-03-10]
    3. Irizarry T, DeVito Dabbs A, Curran CR. Patient portals and patient engagement: a state of the science review. J Med Internet Res 2015 Jun 23;17(6):e148 [FREE Full text] [CrossRef] [Medline]
    4. Posada J, Potvin H, Cookson C. Opening up with Open Notes: writing notes in the era of full patient access. Acad Psychiatry 2021 Jul 26:1-5. [CrossRef] [Medline]
    5. Stults CD, Li J, Frosch DL, Krishnan H, Smith-McCurdy G, Jones VG, et al. Assessment of accuracy and usability of a fee estimator for ambulatory care in an integrated health care delivery network. JAMA Netw Open 2019 Dec 02;2(12):e1917445 [FREE Full text] [CrossRef] [Medline]
    6. Heath S. Patient portal adoption tops 90%, but strong patient use is needed. Patient Engagement HIT.   URL: https://tinyurl.com/352tarjs [accessed 2020-04-29]
    7. Berliner Senderey A, Kornitzer T, Lawrence G, Zysman H, Hallak Y, Ariely D, et al. It's how you say it: systematic A/B testing of digital messaging cut hospital no-show rates. PLoS One 2020 Jun 23;15(6):e0234817 [FREE Full text] [CrossRef] [Medline]
    8. Volpp KG, Asch DA. Make the healthy choice the easy choice: using behavioral economics to advance a culture of health. QJM 2017 May 01;110(5):271-275 [FREE Full text] [CrossRef] [Medline]
    9. Patel MS, Volpp KG, Asch DA. Nudge units to improve the delivery of health care. N Engl J Med 2018 Jan 18;378(3):214-216 [FREE Full text] [CrossRef] [Medline]
    10. Loewenstein G, Brennan T, Volpp KG. Asymmetric paternalism to improve health behaviors. JAMA 2007 Nov 28;298(20):2415-2417. [CrossRef] [Medline]
    11. Asch D, Volpp K. Use behavioral economics to achieve wellness goals. Harvard Business Review. 2014.   URL: https:/​/www.​hhs.gov/​sites/​default/​files/​ocr/​privacy/​hipaa/​understanding/​coveredentities/​hitechact.​pdf [accessed 2022-03-10]
    12. Emanuel EJ, Ubel PA, Kessler JB, Meyer G, Muller RW, Navathe AS, et al. Using behavioral economics to design physician incentives that deliver high-value care. Ann Intern Med 2016 Jan 19;164(2):114-119. [CrossRef] [Medline]
    13. Asch DA, Rosin R. Engineering social incentives for health. N Engl J Med 2016 Dec 29;375(26):2511-2513. [CrossRef] [Medline]
    14. Purnell JQ, Thompson T, Kreuter MW, McBride TD. Behavioral economics: "nudging" underserved populations to be screened for cancer. Prev Chronic Dis 2015 Jan 15;12:E06 [FREE Full text] [CrossRef] [Medline]
    15. Granek B, Evans A, Petit J, James MC, Ma Y, Loper M, et al. Feasibility of implementing a behavioral economics mobile health platform for individuals with behavioral health conditions. Health Technol 2021 Feb 25;11(3):505-510. [CrossRef]
    16. MacCarthy S, Mendoza-Graf A, Wagner Z, L Barreras J, Kim A, Giguere R, et al. The acceptability and feasibility of a pilot study examining the impact of a mobile technology-based intervention informed by behavioral economics to improve HIV knowledge and testing frequency among Latinx sexual minority men and transgender women. BMC Public Health 2021 Feb 12;21(1):341 [FREE Full text] [CrossRef] [Medline]
    17. Graham AK, Munson SA, Reddy M, Neubert SW, Green EA, Chang A, et al. Integrating user-centered design and behavioral science to design a mobile intervention for obesity and binge eating: mixed methods analysis. JMIR Form Res 2021 May 10;5(5):e23809 [FREE Full text] [CrossRef] [Medline]
    18. Wang SY, Groene O. The effectiveness of behavioral economics-informed interventions on physician behavioral change: a systematic literature review. PLoS One 2020 Jun 4;15(6):e0234149 [FREE Full text] [CrossRef] [Medline]
    19. Caturegli G, Materi J, Lombardo A, Milovanovic M, Yende N, Variava E, et al. Choice architecture-based prescribing tool for TB preventive therapy: a pilot study in South Africa. Public Health Action 2020 Sep 21;10(3):118-123 [FREE Full text] [CrossRef] [Medline]
    20. Belli HM, Chokshi SK, Hegde R, Troxel AB, Blecker S, Testa PA, et al. Implementation of a behavioral economics electronic health Record (BE-EHR) module to reduce overtreatment of diabetes in older adults. J Gen Intern Med 2020 Nov 03;35(11):3254-3261 [FREE Full text] [CrossRef] [Medline]
    21. Reed KL, Harvey EM, Everly CJ. The intersection of behavioral economics and the general medicine literature. Am J Med 2021 Nov;134(11):1350-1356.e2 [FREE Full text] [CrossRef] [Medline]
    22. Han H, Gleason KT, Sun C, Miller HN, Kang SJ, Chow S, et al. Using patient portals to improve patient outcomes: systematic review. JMIR Hum Factors 2019 Dec 19;6(4):e15038 [FREE Full text] [CrossRef] [Medline]
    23. Chung S, Lesser LI, Lauderdale DS, Johns NE, Palaniappan LP, Luft HS. Medicare annual preventive care visits: use increased among fee-for-service patients, but many do not participate. Health Aff (Millwood) 2015 Jan;34(1):11-20. [CrossRef] [Medline]
    24. Ganguli I, Souza J, McWilliams JM, Mehrotra A. Trends in use of the US Medicare annual wellness visit, 2011-2014. JAMA 2017 Jun 06;317(21):2233-2235 [FREE Full text] [CrossRef] [Medline]
    25. Lind KE, Hildreth KL, Perraillon MC. Persistent disparities in medicare's annual wellness visit utilization. Med Care 2019 Dec;57(12):984-989. [CrossRef] [Medline]
    26. Bandi P, Minihan AK, Siegel RL, Islami F, Nargis N, Jemal A, et al. Updated review of major cancer risk factors and screening test use in the United States in 2018 and 2019, with a focus on smoking cessation. Cancer Epidemiol Biomarkers Prev 2021 May 19;30(7):1287-1299. [CrossRef]
    27. Tang PC, Ash JS, Bates DW, Overhage JM, Sands DZ. Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption. J Am Med Inform Assoc 2006 Mar 01;13(2):121-126. [CrossRef]
    28. Tang PC, Black W, Buchanan J, Young CY, Hooper D, Lane SR, et al. PAMFOnline: integrating EHealth with an electronic medical record system. AMIA Annu Symp Proc 2003:644-648 [FREE Full text] [Medline]
    29. Behavioral economics: the last mile for health engagement. VALHealth. 2019.   URL: https://tinyurl.com/2p9a397r [accessed 2021-12-14]
    30. Achieving telemedicine adoption using behavioral economics. VALHealth. 2018.   URL: https://tinyurl.com/2dk9j5bk [accessed 2021-12-15]
    31. Increasing engagement with patient portals. VALHealth. 2018.   URL: https://tinyurl.com/5n8e55yw [accessed 2021-12-04]
    32. Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res 2011 May 31;46(3):399-424 [FREE Full text] [CrossRef] [Medline]
    33. Stukel TA, Fisher ES, Wennberg DE, Alter DA, Gottlieb DJ, Vermeulen MJ. Analysis of observational studies in the presence of treatment selection bias: effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. JAMA 2007 Jan 17;297(3):278-285 [FREE Full text] [CrossRef] [Medline]
    34. Ganguli I, Orav EJ, Lupo C, Metlay JP, Sequist TD. Patient and visit characteristics associated with use of direct scheduling in primary care practices. JAMA Netw Open 2020 Aug 03;3(8):e209637 [FREE Full text] [CrossRef] [Medline]
    35. Zhao P, Yoo I, Lavoie J, Lavoie BJ, Simoes E. Web-based medical appointment systems: a systematic review. J Med Internet Res 2017 Apr 26;19(4):e134 [FREE Full text] [CrossRef] [Medline]

    ‎
    OR: odds ratio

    Edited by A Kushniruk; submitted 06.10.21; peer-reviewed by R Krukowski, B Zhao; comments to author 24.10.21; revised version received 17.12.21; accepted 10.01.22; published 30.03.22

    Copyright

    ©Su-Ying Liang, Cheryl D Stults, Veena G Jones, Qiwen Huang, Jeremy Sutton, Guy Tennyson, Albert S Chan. Originally published in JMIR Human Factors (https://humanfactors.jmir.org), 30.03.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on https://humanfactors.jmir.org, as well as this copyright and license information must be included.