Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, December 24 through Wednesday, December 26 inclusive. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 04.09.18 in Vol 5, No 3 (2018): Jul-Sept

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/9875, first published Jan 18, 2018.

This paper is in the following e-collection/theme issue:

    Original Paper

    Designing Emails Aimed at Increasing Family Physicians’ Use of a Web-Based Audit and Feedback Tool to Improve Cancer Screening Rates: Cocreation Process

    1Prevention and Cancer Control, Cancer Care Ontario, Toronto, ON, Canada

    2Institute of Health Policy, Management, and Evaluation, University of Toronto, Toronto, ON, Canada

    3Department of Family and Emergency Medicine, Faculty of Medicine, Laval University, Quebec City, QC, Canada

    4Office of Education and Professional Development, Faculty of Medicine, Laval University, Quebec City, QC, Canada

    5Research Centre of the CHU de Québec, Quebec City, QC, Canada

    6Women’s College Research Institute, Women’s College Hospital, Toronto, ON, Canada

    7Institute for Health System Solutions and Virtual Care, Women's College Hospital, Toronto, ON, Canada

    8Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada

    9School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada

    10School of Psychology, Faculty of Social Sciences, University of Ottawa, Ottawa, ON, Canada

    11Family Practice Health Centre, Women's College Hospital, Toronto, ON, Canada

    12Institute for Clinical Evaluative Sciences, Toronto, ON, Canada

    13Department of Medicine, Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    14Sunnybrook Research Institute, Toronto, ON, Canada

    15Department of Family and Community Medicine, Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    Corresponding Author:

    Noah M Ivers, MD, PhD

    Institute for Health System Solutions and Virtual Care

    Women's College Hospital

    76 Grenville Street

    Toronto, ON, M5S 1B2

    Canada

    Phone: 1 416 323 6060

    Fax:1 416 323 6255

    Email:


    ABSTRACT

    Background: Providing clinical performance data to health professionals, a process known as audit and feedback, can play an important role in health system improvement. However, audit and feedback tools can only be effective if the targeted health professionals access and actively review their data. Email is used by Cancer Care Ontario, a provincial cancer agency, to promote access to a Web-based audit and feedback tool called the Screening Activity Report (SAR); however, current emails that lack behavior change content have been ineffective at encouraging log-in to the SAR.

    Objective: The objective of our study was to describe the process and experience of developing email content that incorporates user input and behavior change techniques (BCTs) to promote the use of the SAR among Ontario primary care providers.

    Methods: Our interdisciplinary research team first identified BCTs shown to be effective in other settings that could be adapted to promote use of the SAR. We then developed draft BCT-informed email content. Next, we conducted cocreation workshops with physicians who had logged in to the SAR more than once over the past year. Participants provided reactions to researcher-developed BCT-informed content and helped to develop an email that they believed would prompt their colleagues to use the SAR. Content from cocreation workshops was brought to focus groups with physicians who had not used the SAR in the past year. We analyzed notes from the cocreation workshops and focus groups to inform decisions about content. Finally, 8 emails were created to test BCT-informed content in a 2×2×2 factorial randomized experiment.

    Results: We identified 3 key tensions during the development of the email that required us to balance user input with scientific evidence, organizational policies, and our scientific objectives, which are as follows: conflict between user preference and scientific evidence, privacy constraints around personalizing unencrypted emails with performance data, and using cocreation methods in a study with the objective of developing an email that featured BCT-informed content.

    Conclusions: Teams tasked with developing content to promote health professional engagement with audit and feedback or other quality improvement tools might consider cocreation processes for developing communications that are informed by both users and BCTs. Teams should be cautious about making decisions solely based on user reactions because what users seem to prefer is not always the same as what works. Furthermore, implementing user recommendations may not always be feasible. Teams may face challenges when using cocreation methods to develop a product with the simultaneous goal of having clearly defined variables to test in later studies. The expected role of users, evidence, and the implementation context all warrant consideration to determine whether and how cocreation methods could help to achieve design and scientific objectives.

    JMIR Hum Factors 2018;5(3):e25

    doi:10.2196/humanfactors.9875

    KEYWORDS



    Introduction

    Audit and feedback is a widely adopted strategy to improve clinical practice guideline adherence [1]. However, audit and feedback tools can only be effective if physicians access them to review their clinical performance data. Physicians who do not see their performance feedback data are not likely to use it for quality improvement [2]. Email is commonly used to communicate with physicians and can be an effective channel for encouraging guideline-recommended care [3] and access to Web-based educational opportunities [4]. Cancer Care Ontario, a provincial government cancer agency, uses email communications to encourage primary care physicians to log in to a Web-based cancer screening tool called the Screening Activity Report (SAR).

    The SAR is a Web-based audit and feedback tool aimed at improving cancer screening-guideline adherence among primary care physicians in Ontario, Canada. The SAR contains patient-level information about rostered patients (ie, patients that are enrolled with a physician) who require action regarding cancer screening (eg, due for screening, overdue for screening, and require follow-up to an abnormal result). To promote regular SAR use, Cancer Care Ontario sends monthly emails to family physicians to inform them that the SAR data has been updated. However, the content of these monthly emails appears to be ineffective at promoting access to the SAR; only 2.37% (129/5445), 3.76% (207/5512), and 4.09% (227/5552) of the SAR-registered physicians logged in to the SAR in February, March, and April 2017, respectively [5]. Additionally, in the same period, less than half of recipients opened the email and only about 7% clicked on the embedded link to access the SAR [6]. Thus, there is a need to understand how to develop emails that are more effective at encouraging family physicians to log in and use the audit and feedback tool.

    There may be a number of reasons why physicians do not open the monthly SAR email, including feeling overwhelmed by the number of emails received and having limited time to open emails [7]. Another potential reason may be that physicians dismiss these emails owing to a lack of compelling content (eg, benefits to the physician and patient of using the SAR). It may be possible to improve SAR access with content that employs a behavioral science approach to address barriers to SAR use.

    In this paper, we describe our process and experience of developing email content involving cocreation workshops and focus group discussions with physician SAR users. The goal of this process was to develop user- and behavior change technique (BCT)-informed email content for a study testing variants of the email [8]. Using BCT classification systems helps to build evidence about which behavior change intervention components work and under what conditions [9].


    Methods

    Design and Setting

    We conducted cocreation workshops and focus groups with family physicians in Toronto and Kingston, Ontario, Canada between January and April 2017. Final email products were tested in a 2×2×2 factorial experiment with Ontario physicians registered for the SAR [8].

    Email Development Process

    We used an iterative process to develop user-informed email content that operationalizes 3 BCTs. Textbox 1 illustrates the steps involved in our email content development process. Throughout the process, we engaged relevant decision makers at Cancer Care Ontario to review content and provide feedback to ensure that the final products would be implementable.


    Textbox 1. Email content development process.
    View this box

    Results

    Participants

    Although we made an effort to recruit adopters for cocreation workshops and nonadopter participants for focus groups, challenges with recruitment required us to be open to the inclusion of adopter participants in the focus groups. Table 1 and Table 2 show the participant characteristics for cocreation workshops and focus groups.

    Behavior Change Techniques

    The full list of BCTs that were initially selected by the research team and considered throughout the project can be found in Multimedia Appendix 1.

    The 3 BCTs to be tested as part of the 2×2×2 factorial randomized experiment include the following: anticipated regret, material incentive (behavior), and problem solving. The final operationalizations of each BCT can be found in Multimedia Appendix 2. The final email with all 3 BCTs can be found in the Multimedia Appendix 3.

    Key Tensions and Resulting Changes to Email Content

    Over the course of the email development process, we used 3 main sources of input to inform decisions about content, including scientific literature, SAR users, and Cancer Care Ontario communication policies. Tensions arose when these sources of input conflicted with each other.

    Conflict Between User Preferences and Scientific Evidence

    Given the evidence that physicians’ decisions are influenced by the desire to minimize regret associated with a potentially wrong decision [15-17], we developed anticipated regret content and rendered it in the form of a reflective question. At first, we considered missing an abnormal result to be the outcome of regret (ie, “How would you feel if you missed an abnormal result?”). However, after discussion with adopters, we refined the anticipated regret content to more explicitly focus on the effect of missing an abnormal test result on patient outcomes (ie, “How would you feel if a patient had a poor outcome because you missed an abnormal test result?”).

    Tensions emerged when users’ reactions conflicted with evidence about the effectiveness of the anticipated regret. Focus group participants expressed feelings of guilt and anger when they read the message and indicated that they did not want to have an emotional response when reading an email. Focus group participants described the anticipated regret content as “confrontational,” “combative,” and “not a nice way of starting.”

    Table 1. Participant characteristics for cocreation workshops.
    View this table
    Table 2. Participant characteristics for focus groups.
    View this table

    One participant restated this message as:

    How would you feel if you were a murderer? How would you feel if you screwed up?
    [SAR Non adopter P019]

    Further, participants articulated that the content implied that they were not doing their job well. For example, one participant stated:

    You know it happens, but you don’t have to sort of start off a positive report that says “we would really like...we think this report is really encouraging you to use it, it can help you with A, B, and C, but you know what really if you missed a test result you’re shitty, you’re really lousy, you’re going to be sued, so use this tool.”...that’s just so condescending, negative on...you know the stress in our life is already there, we don’t need to be told that we feel bad when things go bad.
    [SAR Non adopter P016]

    Exploration of reactions in focus groups revealed that the content, though evoking negative emotions, motivated several participants to read the email further; for example, one participant said:

    I would read more but I would be pissed off.
    [SAR Non adopter P017]

    Another participant acknowledged that the guilt was motivating, aligning with evidence behind the BCT:

    I think the first line is good, how would you feel if patient had a poor outcome because you missed an abnormal test result? I think that’s every doctor’s fear because we go through so many results and we’re making decisions very quickly and it is possible to miss test results, which everybody’s done. And sometimes you get lucky and it’s not a big deal, and sometimes you don’t and you feel horrible about it so I think the guilt thing in there is good, and it’s motivating.
    [SAR Non adopter P011]

    Thus, although we recognized that too much fear can result in physicians avoiding the desired behavior (ie, logging in to the SAR), we believed that these strong negative reactions had the potential to engage recipients and maintain their attention, something that the previous SAR email was seemingly unable to do. We tried to avoid a judgmental tone that would evoke anger and to emphasize in the content that followed the anticipated regret message that logging in to the SAR could mitigate feelings of fear and anxiety about missing critical cancer screening results.

    Participants also had negative reactions to the embedded problem-solving strategies for making time to access the SAR. The problem-solving BCT requires analysis of factors influencing the behavior and selecting strategies that overcome barriers or increase facilitators [11]. Our problem-solving strategies for improved SAR use were informed directly by adopters’ real-world experiences, which included assigning a delegate to access the SAR on their behalf to overcome time pressures. One adopter described why nonadopters such as her colleagues need problem-solving strategies:

    Because it dumbs [accessing the SAR] down, it makes it a stepwise option for them to do. It’s perfect. It’s perfect. That’s what they need. They just need someone to take them by the hand and say, ‘Brian, print this out and give it to Dale and she will book you 15 minutes.’ They can’t find 15 minutes in their life to do this so this is a recipe for following through. It’s really good. I like it a lot.
    [SAR Adopter P008]

    Nevertheless, several focus group participants expressed that they would not use these strategies, often stating that resource and time constraints prevented them from employing them in their practice. In some cases, participants disliked the strategies proposed because they felt that they were unnecessary; for example, one participant said that if they are going to do any clicking, they might as well click into the SAR rather than click into their calendar to make a reminder about logging into the SAR. Moreover, focus group participants described this strategy to book time in their calendar as “patronizing,” “trivial,” and “petty.” For example, one participant stated:

    I think some of the tips are unnecessary. Like, book some calendar time. I know I can do that. I know it’s one of those things everyone should fit in their schedule; however, they can’t. So telling me to put it in your calendar, I’m a grown up.
    [SAR Non adopter P018]

    Though our goal was to provide strategies that helped users overcome time and workload challenges, participants often wanted to eliminate the BCT-informed content and recommended that we “just get to the bottom line- log in to the SAR.”

    Privacy Constraints Around Personalizing Unencrypted Emails With Performance Data

    One significant source of tension occurred when organizational privacy constraints restricted our ability to respond to some content and design changes offered by participants, especially in the case of personalizing content using individualized performance data and embedded performance comparator data.

    When prompted with the following message “10,000 women in Ontario need follow-up to an abnormal pap test; is one of them yours?,” focus group participants wanted the statement to be more personal and reflect the number of their patients that require follow-up:

    Send me an email saying I got 7 abnormal and give me a link to log in. You don’t have to give me patient demographics and names or whatever. Or you can say, ‘we’ve identified patients based on whatever you know, please log in to this to get your confidential list’...you’re sending out an email that you want us to sort of attract physicians, you want to sort of catch them, you want to say. ‘Hey listen, we’ve identified all these abnormal results, they’ve been sitting here in our whatever database.’…notify me that I have 8 patients, that I guarantee you would motivate me to open up and then follow-up but...
    [SAR Non adopter P019]
    Yeah, the fact that I got an abnormal it’s helpful for me to know. You’re helping me because you’re actually letting me know that there’s some abnormals you want to make sure that I am aware of them and you want to make sure it’s like you feel as if your part of a team, ‘Hey we’re sending you this as a reminder that you have some abnormals.’ You sort of feel like it’s not the stick, and the carrot is that you’re actually improving on patient care.
    [SAR Non adopter P017]

    Though participants wanted varying degrees of personalized data (eg, Ontario rates vs individualized follow-up rates), it was clear that personalized data regarding the number of their own patients that required follow-up resonated with participants and had the potential to prompt SAR access.

    In addition to personalized data, participants expressed a desire to have performance comparator data. Adopters described performance comparator information as “definitely the most unique thing” about the SAR in comparison to other tools used to monitor cancer screening in their practice, such as electronic medical records. One adopter expressed feeling comfort in numbers:

    Because you have no sense of [how] the rest of the family physicians in Ontario are functioning...you really do function in a bubble.
    [SAR Adopter P008]

    Similarly, focus group participants found this benefit of the SAR “very interesting” and noted that it sparked curiosity. One participant stated that they were interested in the comparison data from both a medical and legal standpoint:

    ...I am interested in that if I have a glaring difference from other practices, I want to know that from both a medical and a legal point of view.
    [SAR Non adopter P013]

    Although most focus group participants agreed that comparison information was useful, they also reported that the content used to communicate this feature of the SAR was not as compelling as it could be. One participant suggested that the comparator data would be a strong motivator to use the SAR:

    If the data was planted right in the email. Like, this is you, this is people in the LHIN, and this is people in Ontario.
    [SAR Non adopter P012]

    Furthermore, focus group participants thought an image displaying their performance data would motivate them to log in to the SAR, especially if the image showed that they were underperforming in comparison to their peers:

    If there was an image there that I had a gauge of where I was so if I fell below the benchmark I’d probably feel pretty bad about it and I’d want to increase it.
    [SAR Non adopter P012]

    Our research team consulted the Cancer Care Ontario privacy team to understand which data level we could report in the email (eg, Ontario vs individual) and whether we could embed performance data in the content of the email. Performance data are considered personal information, and an error in sending could result in the disclosure of personal information of a physician to an unintended recipient. Given this potential privacy breach, Cancer Care Ontario refrains from sharing any personal information in emails. Thus, content with personal data was not an option. However, reporting aggregate data at the Ontario, regional, or Local Health Integration Network (LHIN)-level was deemed appropriate because it could not be traced back to the performance of an individual physician. Our team decided that LHIN-level data would be the most compelling solution because it was the smallest unit of aggregate data that we could report. We attempted to make the concept about comparison data more salient by associating the use of the SAR for comparison data with social norms: “Thousands of Ontario family doctors access the SAR to compare their screening rates to other family doctors in Ontario and their region.”

    Cocreation Methods Versus Study Objectives

    Our research team has extensive knowledge of behavior change theory and its application; however, at the beginning of this study, we had limited understanding of users’ experience with the SAR and what email content would compel physicians to access the SAR. Accordingly, the research team made great efforts to create conditions for meaningful and substantial user participation in the development of emails. Adopter-generated emails developed during the first workshop yielded descriptions of the SAR and meaningful content about its benefits based on user experiences. However, most content did not explicitly align with a BCT and content that did was often limited to the material incentives BCT; for example, each adopter-generated email referenced how the SAR helps to track or achieve specified levels of preventive care needed to earn their annual Preventive Care Bonus. Moreover, participants often cited comparator information as a major benefit of the SAR, but we could not identify a BCT that accurately reflected this benefit. The research team encountered tensions around the desire to be true to cocreation methods that prioritize user-generated content and the need to accomplish the study objectives of defining, operationalizing, and testing distinct BCTs.

    To address this tension, we used user-generated emails from the first workshop with adopters to act as a starting point for email design and made distinctions between “base content” and “variable content.” Base content was user-generated content that was iteratively refined throughout the process and would remain constant in all emails tested in the experiment; for example, content about how the SAR provides information about patients that have abnormal results and require follow-up was well-received by users and was considered critical information by the research team for all emails to be tested. Variable content, on the other hand, was user-informed BCT content that was created by the research team, focus-tested among participants, and tested in the factorial randomized experiment.

    Iterative content development began with an open-ended cocreative exploration of effective communication about the SAR. During this stage, adopters played a significant role in generating content, especially non-BCT-informed content. However, as the content evolved, study and organizational constraints were sometimes in conflict with user feedback in our quest to have distinct and defined BCTs for testing within an applied setting. These constraints often took precedence, transitioning the role of the participants from content generators to content informants as the project progressed. Further, the role of the research team moved from translators of user ideas and experiences to decision makers, ultimately, as illustrated in the tensions above. This transition of roles was necessary to accomplish the study’s objectives of testing clearly operationalized BCTs.


    Discussion

    Principal Findings

    This study involved the practical application of scientific evidence and methods to the development of emails to promote the use of an audit and feedback tool. We encountered 3 tensions that may be relevant for others who are considering cocreation methods to develop similar communication interventions.

    The conflict between users’ preferences and the broader scientific evidence regarding the potential effectiveness of a BCT highlights that what people seem to prefer and what works are not always the same. BCTs previously shown to be effective may not always elicit positive responses during cocreation or user testing. This tension is not unique to physician users because it also occurred in previous work with patient users relating to the development of mailings for people recovering from acute coronary syndrome [18]. Although users may be “experts of their experience” [19], they are not always experts in how to promote behavior change in themselves or in others [18]. It is necessary for researchers to be thoughtful when making design decisions because it may or may not be appropriate to make decisions solely based on whether user reactions to evidence-based content were positive or negative. It is the role of the researchers and designers to balance user feedback with the broader available evidence and explore the root cause of user reactions during data collection to make purposeful and informed design decisions.

    Organizational constraints, including privacy policies, are a reality of health systems, which may limit the inclusion of personal performance data. Though users frequently recommended embedding personal performance data in the email, which was a finding consistent with best practices in audit and feedback design [20], privacy regulations did not permit our team to include such data in this setting. This created a situation in which users clearly indicated what they believed would help them and yet, we were unable to implement their suggestions.

    Interestingly, the inclusion of individualized performance data may have expanded the scope of the email from an email intended to drive access to (and use of) an audit and feedback tool to an email that could actually be characterized as an audit and feedback intervention itself. There may be an opportunity in a different context with less stringent privacy regulations to provide some data, but not all data, to effectively bait the user to access the audit and feedback tool; for example, the email may present an overview of what kind of data the user could have access to if they were to use the tool. This may be more effective for communications with individuals who are underperformers rather than strong performers because participants noted that data indicating underperformance would motivate action, and research has shown that feedback is most effective when baseline performance is low [21]. However, because we were not able to test this, we do not know if this strategy would, in fact, be effective at driving access to the SAR or other audit and feedback tools.

    Cocreation challenges may have occurred throughout this study because our objective of developing an email with multiple BCTs required substantial knowledge of BCTs. Adopters were able to generate concepts and non-BCT-informed content in the early stages of the research, but their ability to participate in the cocreation of a product with defined research variables such as BCTs was, not surprisingly, limited thereafter. This tension between design goals to develop the best product, service, or tool for a given context and scientific goals to identify or test generalizable concepts or theory has been noted in other contexts [22]. Cocreation methods that invite users to engage in problem solving may be most appropriate in implementation research when the study (and design) objectives are flexible, fluid, and potentially user-driven; for example, cocreation methods may be helpful when engaging users to provide input on the design and functionality of products, such as an educational application aimed at improving knowledge of clinical skills among nursing students [23] or a complex health information system [24]. However, researchers and design teams are likely to face challenges using cocreation methods when products require the application of specific scientific knowledge and should consider the dynamic and changing role of the users from content generators to content informants as the product, service, or tool develops.

    Limitations

    There are several limitations to this study. First, although we started with purposive sampling, we eventually turned to convenience sampling to recruit physicians from Toronto and Kingston. Convenience sampling could potentially contribute to selection bias and failure to recruit physicians with diverse views on how to communicate to physicians about accessing an audit and feedback tool. Furthermore, we did not purposively recruit based on adherence to cancer screening guidelines. Future research with this target audience may consider recruiting based on performance level to understand if reactions to BCT-informed content differ between high performers, average performers, and underperformers. Second, our findings occurred during the development of email communications about a specific audit and feedback tool created to help physicians monitor the cancer screening status of their rostered patients. These findings may or may not apply to the development of products on different communication channels, to the development of products that deal with a different audit and feedback tool, or to a different audience such as specialists rather than family physicians; for example, it may be possible that physicians could spontaneously generate BCT-informed content or products in other contexts. Third, the analysis of the 2×2×2 factorial experiment is currently underway, and we do not yet know the impact of the interventions developed.

    Conclusions

    As audit and feedback tools proliferate across health care systems, there will be an increasing need for effective communications that promote the access and use of such tools. Teams within organizations that are tasked with developing content might consider cocreation methods because they allow for the development of communications that are both user- and BCT-informed. This paper describes a repeatable series of steps that teams could use when pursuing design of implementation interventions, beginning with engaging users to understand what kind of content is compelling to them. Users provide honest feedback about what goes through their mind when they read the content and can provide thoughtful suggestions to inspire developers. However, teams working with users may need to consider whether and how they should balance user feedback with scientific evidence, organizational constraints, and study objectives. Thus, the role of users in decision making and that of BCTs along with the implementation context should all be considered during planning to determine whether cocreation methods would be appropriate to accomplish design objectives.

    Acknowledgments

    This study was conducted with the support of the Ontario Institute for Cancer Research and Cancer Care Ontario through funding provided by the Government of Ontario. The research team would like to acknowledge the Primary Care, Privacy, and Operations teams at Cancer Care Ontario for their help in developing and reviewing content. More specifically, we would like to thank Matthew Jurczak for his diligent review of the content to ensure that it aligns with Cancer Care Ontario’s privacy policies and Petra de Heer for her help to design and implement the emails.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Full list of behaviour change techniques considered during development process and definitions from Michie et al.’s behaviour change technique taxonomy v1[11].

    PDF File (Adobe PDF File), 208KB

    Multimedia Appendix 2

    Operationalizations of behaviour change techniques to be tested in the 2X2X2 factorial experiment and evidence to support testing.

    PDF File (Adobe PDF File), 329KB

    Multimedia Appendix 3

    Final email consisting of all BCT-informed content to be tested in the 2X2X2 factorial randomized experiment.

    PDF File (Adobe PDF File), 558KB

    References

    1. Hysong SJ, Best RG, Pugh JA. Audit and feedback and clinical practice guideline adherence: making feedback actionable. Implement Sci 2006 Apr 28;1:9 [FREE Full text] [CrossRef] [Medline]
    2. Boyce MB, Browne JP, Greenhalgh J. Surgeon's experiences of receiving peer benchmarked feedback using patient-reported outcome measures: a qualitative study. Implement Sci 2014 Jun 27;9:84 [FREE Full text] [CrossRef] [Medline]
    3. Goyder C, Atherton H, Car M, Heneghan CJ, Car J. Email for clinical communication between healthcare professionals. Cochrane Database Syst Rev 2015;2:CD007979. [CrossRef] [Medline]
    4. Abdolrasulnia M, Collins BC, Casebeer L, Wall T, Spettell C, Ray MN, et al. Using email reminders to engage physicians in an Internet-based CME intervention. BMC Med Educ 2004 Sep 29;4:17 [FREE Full text] [CrossRef] [Medline]
    5. CCO, 2017. Reports Accessed by User Report. Unpublished internal dataset. CCO 2017 Jun 01.
    6. CCO, 2017. SAR Refresh Notification MC Reports. Unpublished internal dataset. CCO 2017 May 01.
    7. Badran H, Pluye P, Grad R. Advantages and disadvantages of educational email alerts for family physicians: viewpoint. J Med Internet Res 2015 Feb 27;17(2):e49 [FREE Full text] [CrossRef] [Medline]
    8. Vaisson G, Witteman HO, Bouck Z, Bravo CA, Desveaux L, Llovet D, et al. Testing behavior change techniques to encourage primary care physicians to access cancer screening audit and feedback reports: Protocol for a factorial randomized experiment of email content. JMIR Res Protoc 2018;7(1):e11. [CrossRef]
    9. Gould NJ, Lorencatto F, Stanworth SJ, Michie S, Prior ME, Glidewell L, et al. Application of theory to enhance audit and feedback interventions to increase the uptake of evidence-based transfusion practice: an intervention development protocol. Implement Sci 2014 Jul 29;9:92 [FREE Full text] [CrossRef] [Medline]
    10. Smith WR. Evidence for the effectiveness of techniques To change physician behavior. Chest 2000 Aug;118(2 Suppl):8S-17S. [Medline]
    11. Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med 2013 Aug;46(1):81-95. [CrossRef] [Medline]
    12. Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research. Adm Policy Ment Health 2015 Sep;42(5):533-544 [FREE Full text] [CrossRef] [Medline]
    13. Sedgwick P. Convenience sampling. BMJ 2013 Oct 25;347(oct25 2):f6304-f6304. [CrossRef]
    14. Haggen P, Rowland N, Young G, McKemon S. Smallfire. A Framework for Thinking Through Co-design Workshops   URL: http://www.smallfire.co.nz/wp-content/uploads/2013/11/Co-designworkshopBusstopHandout.pdf [accessed 2018-01-12] [WebCite Cache]
    15. Djulbegovic B, Elqayam S, Reljic T, Hozo I, Miladinovic B, Tsalatsanis A, et al. How do physicians decide to treat: an empirical evaluation of the threshold model. BMC Med Inform Decis Mak 2014 Jun 05;14:47 [FREE Full text] [CrossRef] [Medline]
    16. Coricelli G. The potential role of regret in the physician-patient relationship: insights from neuroeconomics. Adv Health Econ Health Serv Res 2008;20:85-97. [Medline]
    17. Sorum PC, Mullet E, Shim J, Bonnin-Scaon S, Chasseigne G, Cogneau J. Avoidance of anticipated regret: the ordering of prostate-specific antigen tests. Med Decis Making 2004;24(2):149-159. [CrossRef] [Medline]
    18. Witteman HO, Presseau J, Nicholas AE, Jokhio I, Schwalm JD, Grimshaw JM, et al. Negotiating Tensions Between Theory and Design in the Development of Mailings for People Recovering From Acute Coronary Syndrome. JMIR Hum Factors 2017 Mar 01;4(1):e6 [FREE Full text] [CrossRef] [Medline]
    19. Sanders EB, Stappers PJ. Co-creation and the new landscapes of design. CoDesign 2008 Jun 24;4(1):5-18. [CrossRef]
    20. Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ, et al. No more 'business as usual' with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci 2014 Jan 17;9:14 [FREE Full text] [CrossRef] [Medline]
    21. Ivers NM, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012 Jun 13(6):CD000259. [CrossRef] [Medline]
    22. Grocott L, Kobori M. The Affordances of Designing for the Learning Sciences. : Aalto University; 2015 Jun 25 Presented at: Proceedings of the 3rd International Conference for Design Education Researchers; 2015 June 3; Chicago, USA p. 1180-1195   URL: https://tinyurl.com/y999pxup [CrossRef]
    23. O'Connor S, Andrews T. Using Co-Design with Nursing Students to Create Educational Apps for Clinical Training. Stud Health Technol Inform 2016;225:334-338. [Medline]
    24. Kushniruk A, Nøhr C. Participatory Design, User Involvement and Health IT Evaluation. Stud Health Technol Inform 2016;222:139-151. [Medline]
    25. St QT, Brunton JA. Implicit Processes, Self-Regulation, and Interventions for Behavior Change. Front Psychol 2017;8:346 [FREE Full text] [CrossRef] [Medline]
    26. Mostofian F, Ruban C, Simunovic N, Bhandari M. Changing physician behavior: what works? Am J Manag Care 2015 Jan;21(1):75-84 [FREE Full text] [Medline]
    27. Flodgren G, Eccles MP, Shepperd S, Scott A, Parmelli E, Beyer FR. An overview of reviews evaluating the effectiveness of financial incentives in changing healthcare professional behaviours and patient outcomes. Cochrane Database Syst Rev 2011 Jul 06(7):CD009255 [FREE Full text] [CrossRef] [Medline]
    28. Kiran T, Wilton AS, Moineddin R, Paszat L, Glazier RH. Effect of payment incentives on cancer screening in Ontario primary care. Ann Fam Med 2014 Jul;12(4):317-323 [FREE Full text] [CrossRef] [Medline]
    29. Gardner B, Smith L, Lorencatto F, Hamer M, Biddle SJ. How to reduce sitting time? A review of behaviour change strategies used in sedentary behaviour reduction interventions among adults. Health Psychol Rev 2015 Sep 16:1-24. [CrossRef] [Medline]
    30. Kozica S, Lombard C, Teede H, Ilic D, Murphy K, Harrison C. Initiating and continuing behaviour change within a weight gain prevention trial: a qualitative investigation. PLoS One 2015;10(4):e0119773 [FREE Full text] [CrossRef] [Medline]
    31. Samdal GB, Eide GE, Barth T, Williams G, Meland E. Effective behaviour change techniques for physical activity and healthy eating in overweight and obese adults; systematic review and meta-regression analyses. Int J Behav Nutr Phys Act 2017 Dec 28;14(1):42 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    BCT: behavior change technique
    LHIN: local health integration network
    SAR: Screening Activity Report


    Edited by G Eysenbach; submitted 18.01.18; peer-reviewed by TH van de Belt, T Risling, Y van Kasteren; comments to author 15.02.18; revised version received 25.05.18; accepted 18.06.18; published 04.09.18

    ©Caroline A. Bravo, Diego Llovet, Holly O Witteman, Laura Desveaux, Justin Presseau, Marianne Saragosa, Gratianne Vaisson, Shama Umar, Jill Tinmouth, Noah M. Ivers. Originally published in JMIR Human Factors (http://humanfactors.jmir.org), 04.09.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on http://humanfactors.jmir.org, as well as this copyright and license information must be included.