This paper is in the following e-collection/theme issue:

Design and Usability of Consumer Health Tech and Home Monitoring Devices (187) Engagement with and Adherence to Digital Health Interventions, Law of Attrition (457) Web-based and Mobile Health Interventions (3093)

Published on in Vol 9, No 3 (2022): Jul-Sep

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/34925, first published .
The Influencing Contexts and Potential Mechanisms Behind the Use of Web-Based Self-management Support Interventions: Realistic Evaluation

The Influencing Contexts and Potential Mechanisms Behind the Use of Web-Based Self-management Support Interventions: Realistic Evaluation

The Influencing Contexts and Potential Mechanisms Behind the Use of Web-Based Self-management Support Interventions: Realistic Evaluation

Authors of this article:

Marscha Engelen1 Author Orcid Image ;   Betsie van Gaal1, 2 Author Orcid Image ;   Hester Vermeulen1, 2 Author Orcid Image ;   Rixt Zuidema3 Author Orcid Image ;   Sebastian Bredie4 Author Orcid Image ;   Sandra van Dulmen5, 6, 7 Author Orcid Image
Article Authors Cited by Tweetations (3) Metrics

    Original Paper

    1IQ Healthcare, Radboud Institute for Health Sciences, Radboud university medical center, Nijmegen, Netherlands

    2School of Health Studies, HAN University of Applied Sciences, Nijmegen, Netherlands

    3Research Group Proactive Care for Older People, Faculty of Health Care, University of Applied Sciences Utrecht, Utrecht, Netherlands

    4Division of Vascular Medicine, Department of Internal Medicine, Radboud university medical center, Nijmegen, Netherlands

    5Department of Primary and Community Care, Radboud Institute for Health Sciences, Radboud university medical center, Nijmegen, Netherlands

    6Nivel - Netherlands institute for health services research, Department of Communication in healthcare, Utrecht, Netherlands

    7Faculty of Caring Science, Work Life and Social Welfare, University of Borås, Borås, Sweden

    Corresponding Author:

    Marscha Engelen, PhD

    IQ Healthcare

    Radboud Institute for Health Sciences

    Radboud university medical center

    P O Box 9101

    Nijmegen, 6500 HB

    Netherlands

    Phone: 31 630261845

    Email: marscha.engelen@radboudumc.nl


    Background: Self-management can increase self-efficacy and quality of life and improve disease outcomes. Effective self-management may also help reduce the pressure on health care systems. However, patients need support in dealing with their disease and in developing skills to manage the consequences and changes associated with their condition. Web-based self-management support programs have helped patients with cardiovascular disease (CVD) and rheumatoid arthritis (RA), but program use has been low.

    Objective: This study aimed to identify the patient, disease, and program characteristics that determine whether patients use web-based self-management support programs or not.

    Methods: A realistic evaluation methodology was used to provide a comprehensive overview of context (patient and disease characteristics), mechanism (program characteristics), and outcome (program use). Secondary data of adult patients with CVD (n=101) and those with RA (n=77) were included in the study. The relationship between context (sex, age, education, employment status, living situation, self-management [measured using Patient Activation Measure-13], quality of life [measured using RAND 36-item health survey], interaction efficacy [measured using the 5-item perceived efficacy in patient-physician interactions], diagnosis, physical comorbidity, and time since diagnosis) and outcome (program use) was analyzed using logistic regression analyses. The relationship between mechanism (program design, implementation strategies, and behavior change techniques [BCTs]) and outcome was analyzed through a qualitative interview study.

    Results: This study included 68 nonusers and 111 users of web-based self-management support programs, of which 56.4% (101/179) were diagnosed with CVD and 43.6% (78/179) with RA. Younger age and a lower level of education were associated with program use. An interaction effect was found between program use and diagnosis and 4 quality of life subscales (social functioning, physical role limitations, vitality, and bodily pain). Patients with CVD with higher self-management and quality of life scores were less likely to use the program, whereas patients with RA with higher self-management and quality of life scores were more likely to use the program. Interviews with 10 nonusers, 10 low users, and 18 high users were analyzed to provide insight into the relationship between mechanisms and outcome. Program use was encouraged by an easy-to-use, clear, and transparent design and by recommendations from professionals and email reminders. A total of 5 BCTs were identified as potential mechanisms to promote program use: tailored information, self-reporting behavior, delayed feedback, providing information on peer behavior, and modeling.

    Conclusions: This realistic evaluation showed that certain patient, disease, and program characteristics (age, education, diagnosis, program design, type of reminder, and BCTs) are associated with the use of web-based self-management support programs. These results represent the first step in improving the tailoring of web-based self-management support programs. Future research on the interaction between patient and program characteristics should be conducted to improve the tailoring of participants to program components.

    JMIR Hum Factors 2022;9(3):e34925

    doi:10.2196/34925

    Keywords

    self-management; telemedicine; chronic disease; cardiovascular diseases; rheumatoid arthritis; patient dropouts; realistic evaluation; program use 


    Chronic diseases are a major burden for patients, and the growing number of people with (several) chronic conditions puts a strain on our health care systems. The pressure on health care services may be decreased and the quality of life of people with chronic conditions may be improved if these individuals can self-manage their condition and adapt to their situation [1-3]. Self-management is defined as “the individual’s ability to manage the symptoms, treatment, physical consequences, psychological consequences and lifestyle changes inherent in living with a chronic condition” [4]. This is not easy for patients with chronic conditions because they may not feel confident enough to manage their disease [5,6]. Factors such as disease burden, comorbidities, and competing life circumstances can impair a patient’s capacity to self-manage their condition. These obstacles can be overcome with the help of health care professionals, support staff, peers, or digital support programs.

    Self-management support interventions have already been developed for a broad range of long-term medical conditions and have shown improvements in self-management and other health outcomes [7,8]. However, it is challenging to establish self-management support that is feasible for both patients and health care professionals. Web-based self-management support programs may overcome these barriers by providing disease-specific information and personal feedback and by monitoring behavior [9]. Web-based interventions have become more frequent in response to the COVID-19 pandemic, and health care professionals and patients have become more open to digital solutions. Adherence to and uptake of web-based interventions are essential for increasing self-management. However, despite advantages such as easy accessibility and anonymity, studies have shown that the use of and exposure to web-based self-management interventions are unsatisfactory [10,11].

    We recently developed 2 comprehensive, multicomponent, and theory-based web-based self-management interventions using the intervention mapping framework [12]: one for patients with cardiovascular disease (CVD) called Vascular View and one for patients with rheumatoid arthritis (RA) called Coping with RA. Both programs were developed in close collaboration with patients and health care professionals to promote their use and meet patients’ needs. These programs have been described in detail elsewhere [13-16]. Unexpectedly, explorative randomized controlled trials showed no effect of these programs on self-management, possibly because patients were not using them, even though we tried to match them to patients’ needs. Vascular View was used by 62.4% (65/101) of the intervention group and Coping with RA by 63% (50/78) of the intervention group. This phenomenon of participants dropping out of or not using an intervention is called the law of attrition and is a major challenge when developing and evaluating eHealth interventions [17].

    There are many reasons why participants use or do not use a web-based self-management program. Patient characteristics, such as older age, lower education levels, and lower income, have been associated with lower eHealth use [18-20]. Self-management ability, self-efficacy, and quality of life may also be influencing factors, as they are associated with self-management [21,22]. The use of eHealth interventions demands that patients take control of their chronic diseases; therefore, a basic level of self-management is a prerequisite for the use of web-based interventions. Disease characteristics, such as disease burden, may also influence program use. For example, although some patients with CVD do not experience physical symptoms, they still have to adapt their daily routine by making lifestyle changes and taking medication. Patients with CVD might also experience psychosocial consequences, such as being anxious about a secondary cardiovascular event. RA has a more direct physical impact on patients with symptoms such as pain, stiff joints, and fatigue. RA also has psychosocial consequences on patients, such as changes in social roles and feelings of depression. These differences between CVD and RA may influence the self-management needs and program use of these patients. Finally, program characteristics, such as the type of information or applied implementation strategies, may influence whether a patient uses the program [23,24].

    In this study, we identified the patient, disease, and program characteristics that determine whether patients with CVD and patients with RA use the Vascular View and Coping with RA web-based self-management support programs. The findings can be used to tailor web-based self-management support programs to individual patients and thereby increase their use.


    Design

    The realist evaluation methodology was used to gain a comprehensive understanding of why patients use or do not use web-based self-management support interventions [25]. We structured the analysis using the context-mechanism-outcome configuration to identify contextual factors (features of the conditions, eg, patient characteristics, that influence the intervention mechanisms) and potential mechanisms (what and how intervention components are responsible for change) that affect the intervention outcome (Figure 1). The identified mechanisms are described as “potential” as it was beyond the scope of this study to also test their effectiveness. In this study, we used data from 2 previous studies [14,16] that were approved by the medical ethics committee of Arnhem-Nijmegen in the Netherlands (No. 2015-1908 for CVD and 2014-1208 for RA).

    ‎
    Figure 1. Realistic evaluation: context, potential mechanisms, and outcome. PAM: Patient Activation Measure; PEPPI-5: 5-item version perceived efficacy in patient-physician interactions; RAND-36: RAND 36-item health survey.
    View this figure

    Context: Patient and Disease Characteristics

    Overview

    This study included data of 2 patient groups with a chronic disease. Patients in the CVD group had experienced a myocardial infarction, cerebrovascular disease, or peripheral artery disease or a combination of these within 2 months to 1 year of the study starting. Patients in the RA group were diagnosed with RA, a chronic autoimmune disease that predominantly affects the joints, before the start of the study. The baseline data were collected at the start of each study. Inclusion criteria were as follows: (1) aged ≥18 years; (2) ability to read and understand Dutch; (3) access to a computer, internet, and email account; and (4) not receiving psychiatric or psychological treatment.

    Measurements

    The included patient and disease characteristics are expected to be associated with self-management and might, therefore, be related to program use. The following patient characteristics were studied to determine whether they were associated with program use: sex (male or female), age (years), education (low: no education, primary education, or lower secondary education; intermediate: secondary vocational education; and high: higher education or university), work participation (yes or no), living situation (alone or together), self-management, quality of life, and communication efficacy. Self-management was measured using the Patient Activation Measure (PAM-13), which includes statements about an individual’s knowledge, confidence, and skills for self-management of their behavior in response to their chronic illness and about their level of activation. The PAM-13 scores 13 items on a 5-point scale, with a higher score indicating a higher level of patient activation [26,27]. Quality of life was measured using the RAND 36-item health survey (RAND-36), which contains 36 items measuring 8 dimensions: physical functioning, social functioning, physical role limitations, emotional role limitations, mental health, vitality, pain, and general health perception [28]. A higher score indicates a better quality of life. Communication efficacy was measured using the 5-item perceived efficacy in patient-physician interactions, which scores 5 items on a 5-point Likert scale that are summed to determine the total score. A higher score reflects greater confidence in interactions with the health care professional [29,30]. A total of 3 disease characteristics were included: diagnosis (CVD or RA), time since diagnosis (years), and physical comorbidity (yes or no).

    Mechanism: Program Characteristics

    A total of 2 comprehensive, multicomponent, web-based self-management programs were studied for this realistic evaluation: Vascular View and Coping with RA. Multimedia Appendix 1 describes the characteristics of both the programs. There were similarities between the 2 programs and their execution. First, both the programs used the same web-based platform and program design. Second, health care professionals working in one hospital were asked to invite patients to participate in the study. Third, participants had unlimited access to the programs for 12 months between December 2014 and October 2016 and could use the program modules in any sequence and as often as they wanted. A total of 3 program characteristics were considered as potential mechanisms in this realistic evaluation: design, behavior change techniques (BCTs), and implementation strategies [31].

    Vascular View was developed for patients with CVD [13] and contained six modules: (1) coping with CVD and its consequences, (2) setting boundaries in daily life, (3) lifestyle, (4) healthy nutrition, (5) being physically active in a healthy way, and (6) interaction with health professionals. Relevant BCTs (Table 1) were translated to practical applications including general written information on the disease, reading quotes and watching videos of other patients with CVD as role models and receiving personalized feedback, and encouraging participants to write in diaries and perform exercises. Patients filled out a questionnaire to read which modules were recommended for them and received feedback after filling out a lifestyle questionnaire. The implementation strategies were applied in 4 ways. First, the patients received a written instruction manual and digital promotion flyer at the start of the program. Second, they received 1 telephone reminder if they had not used the program within 3 months. Third, they received email reminders if they had started modules but left them incomplete. Finally, a newsletter was sent every 2 months to all participants to informally remind them of the program.

    Coping with RA was developed for patients with RA [15]. The program contained the following nine modules that dealt with health-related problems: (1) balancing rest and activity, (2) setting boundaries, (3) asking for help and support, (4) using medicines, (5) communicating with health professionals, (6) using assistive devices, (7) performing physical exercises, (8) coping with worries, and (9) coping with RA. BCTs (such as providing general information on the disease, self-monitoring, persuasive communication, modeling, self-persuasion, and tailoring) were translated into practical applications (such as texts, videos, exercises, and a medication intake schedule). The content of each program module was tailored to the specific user based on a questionnaire filled out at the start of the web-based program. A total of 3 implementation strategies were applied. First, health care professionals were asked to inform their patients about the web-based program during the consultation. Second, patients received a written instruction manual at the start of the program. Third, patients received biweekly email reminders to use the program.

    Table 1. Overview of applied determinants and behavior change techniques per program.
    DeterminantBehavior change techniquesVascular ViewCoping with RAa
    KnowledgeProvide general information about health behavior✓b✓
    KnowledgeIncrease memory and/or understanding of transferred information✓✓
    AwarenessRisk communication✓✓
    AwarenessSelf-monitoring of behavior✓✓
    AwarenessSelf-report of behaviorN/Ac✓
    AwarenessDelayed feedback of behavior✓N/A
    Social influenceProvide information about peer behavior✓✓
    Social influenceMobilize social norm✓N/A
    AttitudeRe-evaluation of outcomes and self-evaluation✓N/A
    AttitudePersuasive communication✓✓
    AttitudeReward behavioral progressN/A✓
    Self-efficacyModeling✓✓
    Self-efficacyPractice and guided practice✓✓
    Self-efficacyPlan coping responseN/A✓
    Self-efficacyGraded tasks and goal setting✓N/A
    Self-efficacyReattribution training and external attribution of failure✓N/A
    Intention of behaviorGeneral intention formation✓N/A
    Intention of behaviorDevelop medication scheduleN/A✓
    Intention of behaviorSpecific goal setting✓N/A
    Intention of behaviorReview of general and/or specific goals✓N/A
    Intention of behaviorUse of social supportN/A✓
    Action controlUse of cuesN/A✓
    Action controlSelf-persuasionN/A✓
    MaintenanceGoals for maintenance✓N/A

    aRA: rheumatoid arthritis.

    b✓: The behavior change technique was included in the program.

    cN/A: not applicable.

    Outcome: Program Use

    Program use was a dichotomous outcome and was divided into nonusers (0 or 1 visit) and users (≥2 visits). The cut-off point between users and nonusers was arbitrarily set at 2 visits because this was seen as a reflection of whether a patient would have had the opportunity to benefit from the program.

    Analysis

    Relation Between Context and Outcome: Quantitative Analysis

    All quantitative data were analyzed using SPSS Statistics (version 25; IBM Corp). Descriptive analyses were used to describe the patient and disease characteristics of nonusers and users. Differences between the characteristics were tested using 2-tailed t tests and chi-square tests. A 2-sided P value of <.05 was considered statistically significant in all analyses.

    Logistic regression analyses were used to determine which characteristics were associated with program use. Program use (nonuser or user) was the dependent factor. Patient and disease characteristics (sex, age, education, employment status, living situation, self-management, quality of life, interaction efficacy, diagnosis, physical comorbidity, and time since diagnosis) were tested as possible factors. The strength of the relations was interpreted using odds ratios with 95% CIs. Factors with a P value of <.20 were tested in the final model. The model adequacy in the bivariate logistic regression was confirmed with a backward likelihood ratio test. As this is an explorative analysis, the Bonferroni correction was not applied to counteract the problem of multiple variables.

    Sensitivity analysis was performed to compare the characteristics of users and nonusers in the CVD and RA groups. Logistic regression analyses were performed for all characteristics (sex, age, education, employment status, living situation, self-management, quality of life, interaction efficacy, diagnosis, physical comorbidity, and time since diagnosis) and for diagnosis, characteristic, and diagnosis×characteristic. These analyses determined whether there was an interaction between the diagnosis and characteristic. The strengths of the relationships were interpreted using odds ratios with 95% CI.

    Relation Between Mechanism and Outcome: Qualitative Analysis

    As a sequence of efficacy studies of Vascular View and Coping with RA, interviews were conducted to provide insight into (1) why patients used or did not use the web-based program and (2) the experiences with the web-based program among users. The results of the qualitative study on the Coping with RA program have been described elsewhere [23]. In this study, we focused on a part of the interviews to determine potential program characteristics.

    A random selection of Vascular View and Coping with RA users and nonusers were invited for an interview after data on the explorative randomized controlled trials were collected. Purposive sampling was used to select patients regarding the degree to which they used the program. The participants were divided into 3 groups: nonusers, low users, and high users. After providing written consent, each patient was interviewed once via telephone. Semistructured interviews, lasting no longer than 30 minutes, were audio-recorded and anonymized. Interviews were transcribed verbatim and transferred to Excel (Microsoft). A total of 3 themes were determined beforehand: Program design, implementation strategies, and BCTs. The first researcher (ME) thematically analyzed the interviews to identify the potential program characteristics that influence use. First, the verbatim text was read and the relevant parts were marked. Next, the researcher determined barriers and facilitating factors for program use, which were divided into the 3 themes.


    Overview

    We investigated the relations between context, mechanism, and outcome to determine which factors are associated with the use of a web-based self-management support program. Figure 2 summarizes the patient, disease, and program characteristics that influence program use.

    ‎
    Figure 2. Overview of patient and disease characteristics (context) and program characteristics (potential mechanisms) that influence program use (outcome). Underlined variables are factors associated with program use; italicized variables are factors associated with program use in the interaction effect with diagnosis; and the font size reflects the degree of prediction; *P<.20; **P<.05.
    View this figure

    Relation Between Context (Patient and Disease Characteristics) and Outcome (Program Use)

    Descriptive Data

    To analyze the relation between patient and disease characteristics (context) and program use (outcome), 68 patients were defined as nonusers and 111 were defined as users. More users were diagnosed with CVD (63/111, 56.8%) than with RA (48/111, 43.2%). Patient and disease characteristics of the nonuser and user groups are presented in Table 2.

    Table 2. Characteristics of the users and nonusers in the total group, cardiovascular disease (CVD) group, and rheumatoid arthritis (RA) group.
    CharacteristicsTotal groupCVD groupRA group
    Sex (user), n (%)

    		Male59 (59.6)42 (58.3)17 (63.0)

    		Female52 (65)21 (72.4)31 (60.8)
    Level of education (user), n (%)

    		Low21 (77.8)14 (82.4)7 (70.0)

    		Intermediate40 (51.9)16 (47.1)24 (55.8)

    		High50 (66.7)33 (66.0)17 (68)
    Work participation (user), n (%)

    		Yes51 (66.2)24 (60.0)27 (73.0)

    		No60 (58.8)39 (63.9)21 (51.2)
    Living situation (user), n (%)

    		Alone19 (61.3)10 (62.5)9 (60.0)

    		Together92 (62.2)53 (62.4)39 (61.9)
    Physical comorbidity (user), n (%)

    		Yes50 (61.0)24 (60.0)26 (61.9)

    		No61 (62.9)39 (63.9)22 (61.1)
    Age (years), mean (SD)

    		Nonusers64.5 (10.0)a65.1 (9.7)b63.8 (10.5)c

    		Users60.5 (10.4)d61.5 (9.4)e59.2 (11.6)f
    Time since diagnosis (years), mean (SD)

    		Nonusers8.7 (10.6)g5.0 (7.9)h13.4 (11.9)c

    		Users8.1 (10.6)d3.8 (7.7)e13.8 (11.2)f
    Self-management score, PAM-13i, mean (SD)

    		Nonusers40.2 (4.8)j40.7 (4.4)b39.5 (5.4)k

    		Users40.4 (5.5)l40.4 (5.5)e40.3 (5.6)m
    Interaction efficacy, PEPPI-5n, mean (SD)

    		Nonusers21.4 (3.3)j21.3 (2.8)b21.5 (3.8)k

    		Users20.5 (3.3)d20.0 (3.6)e21.1 (2.9)f
    Physical functioning, RAND-36o, mean (SD)

    		Nonusers64.0 (27.8)a71.3 (25.3)b54.3 (28.3)c

    		Users68.8 (25.1)d70.9 (26.0)p66.1 (23.9)f
    Social functioning, RAND-36, mean (SD)

    		Nonusers71.9 (24.1)a77.6 (22.2)b64.6 (24.8)k

    		Users74.9 (22.4)d74.4 (26.0)p75.5 (16.7)f
    Role physical, RAND-36, mean (SD)

    		Nonusers51.1 (43.9)a62.5 (41.0)b36.7 (43.9)k

    		Users56.8 (41.1)d56.7 (41.2)e56.8 (41.5)f
    Role emotional, RAND-36, mean (SD)

    		Nonusers75.1 (40.3)a75.4 (40.0)b74.7 (41.5)c

    		Users80.8 (34.1)d78.8 (35.1)e83.3 (33.0)f
    Mental health, RAND-36, mean (SD)

    		Nonusers75.5 (17.0)a78.1 (17.4)b72.1 (16.1)k

    		Users76.4 (13.9)d75.7 (15.4)e77.4 (11.7)f
    Vitality, RAND-36, mean (SD)

    		Nonusers58.4 (21.2)a62.5 (19.2)b53.1 (22.9)k

    		Users57.5 (18.9)d56.1 (20.4)e59.3 (16.9)f
    Bodily pain, RAND-36, mean (SD)

    		Nonusers70.0 (26.9)a80.2 (23.5)b56.9 (25.5)k

    		Users72.6 (21.8)d75.4 (23.5)e68.9 (19.0)f
    General health, RAND-36, mean (SD)

    		Nonusers51.3 (19.1)a55.4 (18.1)b46.0 (19.4)c

    		Users53.5 (19.0)d53.3 (19.9)e53.8 (17.8)f
    Health change, RAND-36, mean (SD)

    		Nonusers47.8 (22.3)a51.3 (23.2)b43.3 (20.7)k

    		Users51.6 (24.8)d52.4 (25.3)e50.5 (24.5)f

    an=67.

    bn=38.

    cn=29.

    dn=111.

    en=63.

    fn=48.

    gn=66.

    hn=37.

    iPAM-13: Patient Activation Measure.

    jn=68.

    kn=30.

    ln=110.

    mn=47.

    nPEPPI-5: 5-item perceived efficacy in patient-physician interactions.

    oRAND-36: RAND 36-item health survey.

    pn=63.

    Main Analysis of the Relation Between Context and Outcome

    Univariate analyses showed that age, education, and communication efficacy with health care professionals (5-item perceived efficacy in patient-physician interactions) were associated with the use of web-based self-management interventions (Table 3). Table 2 shows that younger patients (mean 60.5, SD 10.4 years) and patients with a lower level of education (21/27, 78% used the intervention) were more likely to use the program than older patients (mean 64.5, SD 10.0 years) and patients with an intermediate level of education (40/77, 52% used the intervention). Furthermore, users scored lower on communication efficacy with health care professionals (mean 20.5, SD 3.3) than nonusers (mean 24.4, SD 3.3). A combination of age and education level provided the best model for predicting the use of the web-based self-management program (Table 4) and correctly predicted whether a person would be a user or nonuser in 69.1% (123/179) of cases. Users were correctly predicted in 91.9% (102/111) of cases and nonusers in 31% (21/68) of cases.

    Table 3. Results of the univariate logistic regressions for all possible factors for total group.

    		ORa (95% CI)P value
    Sex0.79 (0.43-1.46).46
    Age0.96 (0.93-0.99).02b
    Education (reference: low)—intermediate0.31 (0.11-0.85).02b
    Education (reference: low)—high0.57 (0.21-1.60).29
    Employment status1.37 (0.74-2.54).31
    Living situation1.04 (0.47-2.30).93
    Diagnosis0.97 (0.53-1.77).97
    Physical comorbidity0.92 (0.50-1.69).79
    Time since diagnosis1.00 (0.97-1.02).72
    Self-management (PAMc)1.01 (0.95-1.07).77
    Communication efficacy (PEPPId)0.92 (0.83-1.01).08e
    Physical functioning (RAND-36f)1.01 (1.00-1.02).23
    Social functioning (RAND-36)1.01 (0.99-1.02).40
    Role physical (RAND-36)1.00 (1.00-1.01).38
    Role emotional (RAND-36)1.00 (1.00-1.01).32
    Mental health (RAND-36)1.00 (0.98-1.02).68
    Vitality (RAND-36)1.00 (0.98-1.01).78
    Bodily pain (RAND-36)1.01 (0.99-1.02).47
    General health (RAND-36)1.01 (0.99-1.02).47
    Health change (RAND-36)1.01 (0.99-1.02).30

    aOR: odds ratio.

    bP<.05.

    cPAM: Patient Activation Measure.

    dPEPPI-5: 5-item perceived efficacy in patient-physician interactions.

    eP<.20.

    fRAND-36: RAND 36-item health survey.

    Table 4. Final model of factors associated with the use of web-based self-management programsa.

    		BSEORb (95% CI)P value
    Constant3.581.16N/Ac.002
    Age−0.040.0170.96 (0.93-1.00).03
    Education (intermediate vs low)−1.060.520.35 (0.12-0.96).04

    aNagelkerke R2=0.049.

    bOR: odds ratio.

    cN/A: not applicable.

    Sensitivity Analysis of the Relation Between Context and Outcome

    Sensitivity analysis showed a significant interaction between diagnosis and the RAND-36 subscales social functioning, physical role limitations, vitality, and bodily pain (Table 5). The descriptive data presented in Table 2 show that scores on self-management (PAM-13) and some quality-of-life subscales (RAND-36) were different between the CVD and RA groups. Patients with CVD with higher scores on self-management and quality of life were less likely to use the program. In contrast, patients with RA with higher scores on self-management and quality of life were more likely to use the program.

    Table 5. Results of the interaction effects between diagnosis (cardiovascular disease and rheumatoid arthritis) and possible factors.

    		ORa (95% CI)P value
    Sex2.06 (0.54-7.89).29
    Age1.00 (0.94-1.07).95
    Education (reference: low)—intermediate2.84 (0.37-22.06).32
    Education (reference: low)—high2.19 (0.27-17.98).47
    Employment status3.04 (0.87-10.66).08b
    Living situation1.09 (0.22-5.37).92
    Physical comorbidity1.22 (0.36-4.18).75
    Time since diagnosis1.02 (0.96-1.09).50
    Self-management (PAMc)1.04 (0.93-1.17).52
    Communication efficacy (PEPPId)1.09 (0.90-1.33).39
    Physical functioning (RAND-36e)1.02 (0.99-1.04).14b
    Social functioning (RAND-36)1.03 (1.00-1.06).03f
    Role physical (RAND-36)1.02 (1.00-1.03).05f
    Role emotional (RAND-36)1.00 (0.99-1.02).64
    Mental health (RAND-36)1.04 (1.00-1.08).08b
    Vitality (RAND-36)1.03 (1.00-1.07).04f
    Bodily pain (RAND-36)1.04 (1.01-1.06).02f
    General health (RAND-36)1.03 (1.00-1.07).09b
    Health change (RAND-36)1.01 (0.99-1.04).37

    aOR: odds ratio.

    bP<.20.

    cPAM: Patient Activation Measure.

    dPEPPI: Perceived Efficacy in Patient-Physician Interactions.

    eRAND-36: RAND 36-item health survey.

    fP<.05.

    Relation Between Mechanisms (Program Design, Implementation Strategies, and BCTs) and Outcome (Program Use)

    A random sample of study participants was interviewed to gain insight into why they did or did not use the web-based self-management program. In the CVD group, 6 nonusers, 6 low users, and 6 high users were interviewed. In the RA group, 4 nonusers, 4 low users, and 13 high users were interviewed. The results were divided into 3 themes: program design, implementation strategies, and BCTs. Table 6 provides quotes that show the barriers and facilitators for program use on the 3 themes: program design, implementation strategies, and BCTs.

    Most interviewees were pleased with the program design. However, some experienced difficulties in using the program, and so they did not use it as often. A search function would make it easier to find relevant information. Several users and nonusers stated that they had overlooked parts of the program; for example, 1 participant only used the diaries because he did not know that training modules were available. Another major reason for not using the program were problems with logging in. These observations indicate that ease of use was an important factor for program use among our respondents.

    Explanations given by the respondents as to why they did or did not use the program also revealed factors affecting program use. Several respondents stated that they did not participate for their own benefit but rather to facilitate scientific research. Others used the program following advice from their health care professional or because they were curious and wanted to better understand their disease. The biweekly reminders to fill out the diaries in the Coping with RA program helped many respondents to use the diaries.

    Table 6. Quotes from the interviews with users and nonusers.

    		BarriersFacilitators
    Program design“Well, I couldn’t log in. Somehow I really couldn’t, or it wasn’t clear to me. Through the internet I find it very difficult to do.” (Coping with RAa, participant 5)
    • “Yes I liked the lay-out. The information was orderly, you could easily click on what you wanted to see. So the program was very well organized.” (Coping with RA, participant 21)
      • ‎
    Implementation strategiesBarriers for implementation were not described.
    • “The hospital nurse advised me to use the program.” (Vascular View, participant 1)
      • ‎
    • “If I received an email that said I still had something to do, I always did.” (Coping with RA, participant 8)
      • ‎
    Behavior change techniques“The program only gives input but I missed feedback options, for example to keep track of my weight.” (Vascular View, participant 16)
    • “I wanted information on how to deal with my recent diagnosis.” (Vascular View, participant 7)
      • ‎

    aRA: rheumatoid arthritis.

    Comments related to program content were assigned to the relevant BCTs, and some of these BCTs were identified as potential mechanisms affecting program use. The first BCT (providing general information about health behavior) was often mentioned in the interviews. For example, respondents with a long disease history stated that the information was too general. Furthermore, some respondents saw on the overview page that none of the modules contained new or interesting information, and so they did not use the program further. Respondents reported that reliable information was a reason for using the program. The Vascular View program includes a physical activity and nutrition diary (for the self-monitoring of behavior BCT), which was rarely used. One respondent said they had missed a feedback function in the diaries and had already used other, more advanced, mobile apps instead. The pain and fatigue diaries in the Coping with RA program were used more often by respondents (for the self-report of behavior BCT). Patients appreciated the possibility of keeping track of their pain and fatigue and of receiving a graphical overview of their input (the delayed feedback of behavior BCT). Program users also liked the stories and videos of peers (which provided information about peer behavior BCT and modeling BCT). One respondent said that these made her feel recognized and supported and showed her that she was on the right track.


    Principal Findings

    In this realistic evaluation of 2 web-based self-management interventions, we searched for patient, disease, and program characteristics that determine whether patients will use the programs. Regarding the relationship between context and outcome, patient and disease characteristics, younger age, and lower level of education were associated with program use. In addition, 4 quality of life subscales (social function, physical role limitations, vitality, and bodily pain) interacted significantly with the diagnosis group to affect program use. Regarding the relationship between Potential Mechanisms (program characteristics) and outcome, participants indicated that an easy-to-use, clear, and transparent design would motivate them to use the program. Email reminders and recommendations from health care professionals were found to be potential implementation mechanisms for promoting program use. The top five BCT techniques that encouraged interviewees to use the program were (1) tailored information, (2) self-report of behavior, (3) delayed feedback, (4) information about peer behavior, and (5) modeling.

    Tailoring Web-Based Self-management Interventions to Increase Program Use

    Our findings show that patient and disease characteristics can be used to tailor web-based self-management interventions and, therefore, increase their use. Younger age increased program use in our study, which is in agreement with the results of previous studies. However, in contrast to our finding that a lower level of education increased program use, earlier studies showed that a higher level of education increased program use [18-20,32,33]. Despite this discrepancy, these results show that age and education both influence program use, possibly because they are both related to eHealth literacy. eHealth literacy is the ability to seek, find, understand, and appraise health information from electronic sources and to use this knowledge to address or solve a health problem [34]. Concerns have been raised about a digital divide, which is the gap between patients who are able to use eHealth and those who are not [19]. Our study emphasizes the need to pay attention to these issues, as both age and education are strongly related to eHealth literacy [35], and eHealth literacy is needed to benefit from web-based interventions. Different forms of self-management support should be provided to people with low eHealth literacy.

    Disease burden can be both mental and physical and is another possible factor related to the use of web-based self-management support programs. Patients with RA have a lower physical quality of life and experience more pain than those with CVD. Individuals with episodic or deteriorating diseases such as RA have different self-management support needs than those with stable chronic diseases [36,37]. Patients with CVD have reported fewer self-management support needs than those with other chronic diseases because their disease as a smaller impact on their live [38]. These variations in the perceived burden of disease can affect the motivation to change. A higher perceived disease burden has been associated with a higher perception of the necessity for treatment, which increases adherence to treatment [39]. We have shown this in this study; in the RA group, users rated their physical quality of life as higher than nonusers, whereas in the CVD group, users rated their physical quality of life as lower than nonusers. This suggests that a certain level of burden is needed to feel urgency and to be motivated to use a web-based self-management support program. However, a web-based intervention might not be sufficient when the disease burden is too high. In such cases, face-to-face support from health care professionals is recommended [23].

    The Influence of Program Characteristics

    The study participants provided some recommendations for an effective web-based self-management support program. These recommendations included being easy to use, providing appropriate reminders, tailoring information to the user, allowing patients to self-report their behavior and receive delayed feedback, and providing information about peer behavior and modeling. These results are in line with those of a Delphi study that identified new information and the possibility of monitoring personal progress as important factors promoting the use of an eHealth self-management intervention [40]. In addition, previous research has shown that peer support and email or phone contact increase the use of eHealth interventions [10]. These observations suggest that adding an interactive component to our Vascular View and Coping with RA programs, which allows users to communicate with peers and health care professionals, may promote program use. Counselor support has been found to be important for program use in previous studies [10], and our participants stated that interaction would have stimulated them to use the program. The role of health care professionals should never be underestimated, especially as blended care (a combination of eHealth interventions and face-to-face consultations with a health care professional) increases the use of eHealth interventions, including more program components [41].

    One Size Does Not Fit All

    Our results emphasize that one program will not be suitable for every patient. Self-management programs should be tailored to patients’ individual needs. It should also be noted that not all patients can use and benefit from web-based interventions. The validated Self-Management Screening (SeMaS) questionnaire can help identify potential barriers to self-management and can help health care professionals determine their patients’ support needs [42]. The factors affecting program use identified in this study were in accordance with the components of the SeMaS, including age, education, disease burden (both low and high disease burden can be barriers to self-management), computer skills, and social support. The SeMaS can help health care professionals to choose appropriate interventions and to decide which patients would benefit from a web-based self-management support intervention [43].

    The Use of a Realistic Evaluation

    Given the complexity of web-based self-management interventions, realistic evaluations can reveal what makes an intervention work, which a simple cause-and-effect relationship between an intervention and its outcome may not be able to do. This is especially important for eHealth interventions because dropout and nonuse rates are high [17]. The aim of a realistic evaluation is to determine what works for whom, in what circumstances, and why. We tried to answer these questions by analyzing what patient and disease characteristics influence program use (context) and by describing what program characteristics influence program use (potential mechanisms; Figure 2). Unfortunately, we were not able to analyze the interaction between context and mechanism and how this affects the outcome. This should be addressed in future research to further improve the tailoring and effectiveness of eHealth interventions.

    Limitations

    The findings of our realistic evaluation should be considered in the context of several limitations. The principal limitation was that we used retrospective data collected in 2 separate studies. However, both studies were conducted by the same research group and had the same study design. It was already decided in the development phase that the data would be merged for an overarching study; however, we could not include more questions about factors related to program use in the questionnaires. The interviews were conducted to retrieve patients’ experiences, not to identify program characteristics that influence program use. Vascular View and Coping with RA were developed based on BCTs, and most of these were unobtrusively included in the program. In addition, Vascular View and Coping with RA applied different implementation strategies that could have influenced program use, making the programs harder to compare. Therefore, the program characteristics identified in this study are potential mechanisms and should be tested in future research. Another limitation was that physical comorbidity and time since diagnosis were measured using a questionnaire. Although this provides insight into patients’ experiences, the self-reporting of clinical variables is not always reliable. The last limitation of this study was that we only included participants who had access to the internet and an email address. This may have biased our results by excluding people with a very low level of eHealth literacy. However, the internet is easily accessible in the Netherlands (97% of households have access to the internet [44]), so most people would have been able to participate.

    Conclusions

    This realistic evaluation identified contexts and potential mechanisms, in the form of patient, disease, and program characteristics, that are associated with the use of web-based self-management support programs. Our results emphasized the importance of (1) tailoring interventions to patients’ needs (depending on age, education, and program characteristics) to increase program use and (2) considering whether all patients can use eHealth interventions (depending on disease burden and eHealth literacy) and providing alternative self-management support when needed. These results are a first step toward improving the tailoring and use of web-based self-management support programs. Future research into the interaction between patient and program characteristics and how this affects program use should be conducted to improve the tailoring of participants to program components.

    Acknowledgments

    This study was funded by ZonMw, the Netherlands Organization for Health Research and Development (grant 520001001). The authors would like to thank all the patients with cardiovascular disease (CVD) and rheumatoid arthritis (RA) who contributed to this study.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Description of the web-based self-management support programs Vascular View and Coping with Rheumatoid Arthritis.

    DOCX File , 13 KB
    1. Bodenheimer T, Wagner EH, Grumbach K. Improving primary care for patients with chronic illness. JAMA 2002 Oct 09;288(14):1775-1779. [CrossRef] [Medline]
    2. Clark NM. Management of chronic disease by patients. Annu Rev Public Health 2003;24:289-313. [CrossRef] [Medline]
    3. Barker I, Steventon A, Williamson R, Deeny SR. Self-management capability in patients with long-term conditions is associated with reduced healthcare utilisation across a whole health economy: cross-sectional analysis of electronic health records. BMJ Qual Saf 2018 Dec;27(12):989-999 [FREE Full text] [CrossRef] [Medline]
    4. Barlow J, Wright C, Sheasby J, Turner A, Hainsworth J. Self-management approaches for people with chronic conditions: a review. Patient Educ Couns 2002;48(2):177-187. [CrossRef] [Medline]
    5. Barnason S, Zimmerman L, Nieveen J, Schulz P, Young L. Patient recovery and transitions after hospitalization for acute cardiac events: an integrative review. J Cardiovasc Nurs 2012;27(2):175-191. [CrossRef] [Medline]
    6. Egberg L, Andreassen S, Mattiasson A. Experiences of living with intermittent claudication. J Vasc Nurs 2012 Mar;30(1):5-10. [CrossRef] [Medline]
    7. Lorig KR, Sobel DS, Ritter PL, Laurent D, Hobbs M. Effect of a self-management program on patients with chronic disease. Eff Clin Pract 2001;4(6):256-262. [Medline]
    8. Coleman MT, Newton KS. Supporting self-management in patients with chronic illness. Am Fam Physician 2005 Oct 15;72(8):1503-1510 [FREE Full text] [Medline]
    9. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010 Feb 17;12(1):e4 [FREE Full text] [CrossRef] [Medline]
    10. Brouwer W, Kroeze W, Crutzen R, de Nooijer J, de Vries NK, Brug J, et al. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med Internet Res 2011 Jan 06;13(1):e2 [FREE Full text] [CrossRef] [Medline]
    11. Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JE. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012 Nov 14;14(6):e152 [FREE Full text] [CrossRef] [Medline]
    12. Planning Health Promotion Programs An Intervention Mapping Approach. Hoboken, New Jersey, United States: Wiley; 2016.
    13. Puijk-Hekman S, van Gaal BG, Bredie SJ, Nijhuis-van der Sanden MW, van Dulmen S. Self-management support program for patients with cardiovascular diseases: user-centered development of the tailored, web-based program vascular view. JMIR Res Protoc 2017 Feb 08;6(2):e18 [FREE Full text] [CrossRef] [Medline]
    14. Engelen MM, van Dulmen S, Puijk-Hekman S, Vermeulen H, Nijhuis-van der Sanden MW, Bredie SJ, et al. Evaluation of a web-based self-management program for patients with cardiovascular disease: explorative randomized controlled trial. J Med Internet Res 2020 Jul 24;22(7):e17422 [FREE Full text] [CrossRef] [Medline]
    15. Zuidema RM, van Gaal BG, van Dulmen S, Repping-Wuts H, Schoonhoven L. An online tailored self-management program for patients with rheumatoid arthritis: a developmental study. JMIR Res Protoc 2015 Dec 25;4(4):e140 [FREE Full text] [CrossRef] [Medline]
    16. Zuidema R, van Dulmen S, Nijhuis-van der Sanden M, Meek I, van den Ende C, Fransen J, et al. Efficacy of a web-based self-management enhancing program for patients with rheumatoid arthritis: explorative randomized controlled trial. J Med Internet Res 2019 Apr 30;21(4):e12463 [FREE Full text] [CrossRef] [Medline]
    17. Eysenbach G. The law of attrition. J Med Internet Res 2005 Mar 31;7(1):e11 [FREE Full text] [CrossRef] [Medline]
    18. Richtering SS, Hyun K, Neubeck L, Coorey G, Chalmers J, Usherwood T, et al. eHealth literacy: predictors in a population with moderate-to-high cardiovascular risk. JMIR Hum Factors 2017 Jan 27;4(1):e4 [FREE Full text] [CrossRef] [Medline]
    19. Reiners F, Sturm J, Bouw LJ, Wouters EJ. Sociodemographic factors influencing the use of eHealth in people with chronic diseases. Int J Environ Res Public Health 2019 Feb 21;16(4):645 [FREE Full text] [CrossRef] [Medline]
    20. Hardiker NR, Grant MJ. Factors that influence public engagement with eHealth: a literature review. Int J Med Inform 2011 Jan;80(1):1-12. [CrossRef] [Medline]
    21. Choi E, Park J, Min D, Lee HS, Ahn J. Association between self-management behaviour and quality of life in people with heart failure: a retrospective study. BMC Cardiovasc Disord 2022 Mar 08;22(1):90 [FREE Full text] [CrossRef] [Medline]
    22. Heijmans M, Waverijn G, Rademakers J, van der Vaart R, Rijken M. Functional, communicative and critical health literacy of chronic disease patients and their importance for self-management. Patient Educ Couns 2015 Jan;98(1):41-48. [CrossRef] [Medline]
    23. Zuidema RM, Van Dulmen S, Nijhuis-van der Sanden MW, Fransen J, Van Gaal BG. qLessons learned from patients with access to an online self-management enhancing program for RA patients: qualitative analysis of interviews alongside a randomized clinical trial. Patient Educ Couns 2019 Jun;102(6):1170-1177. [CrossRef] [Medline]
    24. Varsi C, Solberg Nes L, Kristjansdottir OB, Kelders SM, Stenberg U, Zangi HA, et al. Implementation strategies to enhance the implementation of eHealth programs for patients with chronic illnesses: realist systematic review. J Med Internet Res 2019 Sep 27;21(9):e14255 [FREE Full text] [CrossRef] [Medline]
    25. Realistic Evaluation. London: SAGE Publications; 1997.
    26. Hibbard JH, Mahoney ER, Stockard J, Tusler M. Development and testing of a short form of the patient activation measure. Health Serv Res 2005 Dec;40(6 Pt 1):1918-1930 [FREE Full text] [CrossRef] [Medline]
    27. Rademakers J, Nijman J, van der Hoek L, Heijmans M, Rijken M. Measuring patient activation in The Netherlands: translation and validation of the American short form Patient Activation Measure (PAM13). BMC Public Health 2012 Jul 31;12:577 [FREE Full text] [CrossRef] [Medline]
    28. VanderZee KI, Sanderman R, Heyink JW, de Haes H. Psychometric qualities of the RAND 36-Item Health Survey 1.0: a multidimensional measure of general health status. Int J Behav Med 1996 Jun;3(2):104-122. [CrossRef] [Medline]
    29. Maly RC, Frank JC, Marshall GN, DiMatteo MR, Reuben DB. Perceived efficacy in patient-physician interactions (PEPPI): validation of an instrument in older persons. J Am Geriatr Soc 1998 Jul;46(7):889-894. [CrossRef] [Medline]
    30. ten Klooster PM, Oostveen JC, Zandbelt LC, Taal E, Drossaert CH, Harmsen EJ, et al. Further validation of the 5-item Perceived Efficacy in Patient-Physician Interactions (PEPPI-5) scale in patients with osteoarthritis. Patient Educ Couns 2012 Apr;87(1):125-130. [CrossRef] [Medline]
    31. Coding manual for behavior change techniques. OSF. 2007.   URL: http:///C:/Users/user/Downloads/Coding_Manual_Taxonomy_Diabetes.pdf [accessed 2021-10-21]
    32. Kontos E, Blake KD, Chou WS, Prestin A. Predictors of eHealth usage: insights on the digital divide from the Health Information National Trends Survey 2012. J Med Internet Res 2014 Jul 16;16(7):e172 [FREE Full text] [CrossRef] [Medline]
    33. Ali MA, Alam K, Taylor B, Ashraf M. Examining the determinants of eHealth usage among elderly people with disability: the moderating role of behavioural aspects. Int J Med Inform 2021 May;149:104411. [CrossRef] [Medline]
    34. Norman CD, Skinner HA. eHealth literacy: essential skills for consumer health in a networked world. J Med Internet Res 2006 Jun 16;8(2):e9 [FREE Full text] [CrossRef] [Medline]
    35. Hsu W. The effect of age on electronic health literacy: mixed-method study. JMIR Hum Factors 2019 Apr 21;6(2):e11480 [FREE Full text] [CrossRef] [Medline]
    36. van Houtum L, Rijken M, Heijmans M, Groenewegen P. Self-management support needs of patients with chronic illness: do needs for support differ according to the course of illness? Patient Educ Couns 2013 Dec;93(3):626-632. [CrossRef] [Medline]
    37. Been-Dahmen JM, Walter MJ, Dwarswaard J, Hazes JM, van Staa A, Ista E. What support is needed to self-manage a rheumatic disorder: a qualitative study. BMC Musculoskelet Disord 2017 Feb 16;18(1):84 [FREE Full text] [CrossRef] [Medline]
    38. Huygens MW, Vermeulen J, Swinkels IC, Friele RD, van Schayck OC, de Witte LP. Expectations and needs of patients with a chronic disease toward self-management and eHealth for self-management purposes. BMC Health Serv Res 2016 Jul 08;16:232 [FREE Full text] [CrossRef] [Medline]
    39. Berglund E, Lytsy P, Westerling R. Adherence to and beliefs in lipid-lowering medical treatments: a structural equation modeling approach including the necessity-concern framework. Patient Educ Couns 2013 Apr;91(1):105-112 [FREE Full text] [CrossRef] [Medline]
    40. Brouwer W, Oenema A, Crutzen R, de Nooijer J, de Vries NK, Brug J. An exploration of factors related to dissemination of and exposure to internet-delivered behavior change interventions aimed at adults: a Delphi study approach. J Med Internet Res 2008 Apr 16;10(2):e10 [FREE Full text] [CrossRef] [Medline]
    41. Talboom-Kamp EP, Verdijk NA, Kasteleyn MJ, Numans ME, Chavannes NH. From chronic disease management to person-centered eHealth; a review on the necessity for blended care. Clin eHealth 2018 Mar;1(1):3-7. [CrossRef]
    42. Eikelenboom N, Smeele I, Faber M, Jacobs A, Verhulst F, Lacroix J, et al. Validation of Self-Management Screening (SeMaS), a tool to facilitate personalised counselling and support of patients with chronic diseases. BMC Fam Pract 2015 Nov 11;16:165 [FREE Full text] [CrossRef] [Medline]
    43. Eikelenboom N, van Lieshout J, Wensing M, Smeele I, Jacobs AE. Implementation of personalized self-management support using the self-management screening questionnaire SeMaS; a study protocol for a cluster randomized trial. Trials 2013 Oct 17;14:336 [FREE Full text] [CrossRef] [Medline]
    44. The Netherlands on the European Scale. CBS. 2019.   URL: https://longreads.cbs.nl/european-scale-2019/internet/ [accessed 2021-10-21]

    ‎
    BCT: behavior change technique
    CVD: cardiovascular disease
    PAM-13: Patient Activation Measure
    RA: rheumatoid arthritis
    RAND-36: RAND 36-item health survey
    SeMaS: Self-Management Screening

    Edited by A Kushniruk; submitted 12.11.21; peer-reviewed by R Grainger, H Ayatollahi; comments to author 10.01.22; revised version received 27.03.22; accepted 14.05.22; published 01.07.22

    Copyright

    ©Marscha Engelen, Betsie van Gaal, Hester Vermeulen, Rixt Zuidema, Sebastian Bredie, Sandra van Dulmen. Originally published in JMIR Human Factors (https://humanfactors.jmir.org), 01.07.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on https://humanfactors.jmir.org, as well as this copyright and license information must be included.