This paper is in the following e-collection/theme issue:

Health Risk and Perception in Cognitive Reasoning (17) Mobile Health (mhealth) (2939) Usability of Apps and User Perceptions of mHealth (974) Innovations and Technology for Healthy Eating Education (414) mHealth for Wellness, Behavior Change and Prevention (2871) Usability Evaluation Case Studies (191) Cardiovascular Disease Prevention (137) Focus Groups and Qualitative Research for Human Factors Research (851) Quality Improvement (253) Design and Usability of Consumer Health Tech and Home Monitoring Devices (218)

Published on in Vol 9, No 2 (2022): Apr-Jun

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/34704, first published .
A Web-Based Health Application to Translate Nutrition Therapy for Cardiovascular Risk Reduction in Primary Care (PortfolioDiet.app): Quality Improvement and Usability Testing Study

A Web-Based Health Application to Translate Nutrition Therapy for Cardiovascular Risk Reduction in Primary Care (PortfolioDiet.app): Quality Improvement and Usability Testing Study

A Web-Based Health Application to Translate Nutrition Therapy for Cardiovascular Risk Reduction in Primary Care (PortfolioDiet.app): Quality Improvement and Usability Testing Study

Authors of this article:

Meaghan E Kavanagh1, 2, 3 Author Orcid Image ;   Laura Chiavaroli1, 2, 3 Author Orcid Image ;   Andrea J Glenn1, 2, 3, 4 Author Orcid Image ;   Genevieve Heijmans3 Author Orcid Image ;   Shannan M Grant5, 6, 7 Author Orcid Image ;   Chi-Ming Chow8 Author Orcid Image ;   Robert G Josse1, 2, 3, 9, 10 Author Orcid Image ;   Vasanti S Malik1, 4 Author Orcid Image ;   William Watson11 Author Orcid Image ;   Aisha Lofters12, 13, 14 Author Orcid Image ;   Candice Holmes11 Author Orcid Image ;   Julia Rackal11 Author Orcid Image ;   Kristie Srichaikul2, 3 Author Orcid Image ;   Diana Sherifali15 Author Orcid Image ;   Erna Snelgrove-Clarke16 Author Orcid Image ;   Jacob A Udell17, 18, 19, 20, 21, 22 Author Orcid Image ;   Peter Juni9, 18, 19 Author Orcid Image ;   Gillian L Booth19, 20, 21, 23 Author Orcid Image ;   Michael E Farkouh22 Author Orcid Image ;   Lawrence A Leiter1, 2, 3, 9, 10, 21 Author Orcid Image ;   Cyril W C Kendall1, 2, 3, 24 Author Orcid Image ;   David J A Jenkins1, 2, 3, 9, 10, 21 Author Orcid Image ;   John L Sievenpiper1, 2, 3, 9, 10, 21 Author Orcid Image
Article Authors Cited by (7) Tweetations (16) Metrics

    Original Paper

    1Department of Nutritional Sciences, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    2Clinical Nutrition and Risk Factor Modification Center, St. Michael’s Hospital, Unity Health Toronto, Toronto, ON, Canada

    3Toronto 3D Knowledge Synthesis and Clinical Trials Unit, St. Michael’s Hospital, Unity Health Toronto, Toronto, ON, Canada

    4Department of Nutrition, Harvard T.H. Chan School of Public Health, Boston, MA, United States

    5Department of Applied Human Nutrition, Mount Saint Vincent University, Halifax, NS, Canada

    6Departments of Pediatrics and Obstetrics and Gynaecology, IWK Health, Halifax, NS, Canada

    7Department of Obstetrics and Gynaecology, Faulty of Medicine, Dalhousie University, Halifax, NS, Canada

    8Division of Cardiology, St. Michael’s Hospital, Unity Health Toronto, Toronto, ON, Canada

    9Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Unity Health Toronto, Toronto, ON, Canada

    10Division of Endocrinology and Metabolism, St. Michael’s Hospital, Unity Health Toronto, Toronto, ON, Canada

    11Department of Family and Community Medicine, St. Michael’s Hospital, Unity Health Toronto, Toronto, ON, Canada

    12Department of Family and Community Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    13Family Practice Health Centre, Women’s College Hospital, Toronto, ON, Canada

    14Peter Gilgan Centre for Women’s Cancers, Women’s College Hospital, Toronto, ON, Canada

    15School of Nursing, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada

    16School of Nursing, Faculty of Health Sciences, Queen's University, Kingston, ON, Canada

    17Women’s College Research Institute and Cardiovascular Division, Department of Medicine, Women’s College Hospital, University of Toronto, Toronto, ON, Canada

    18Applied Health Research Centre, Li Ka Shing Knowledge Institute of St. Michael's Hospital, Department of Medicine, University of Toronto, Toronto, ON, Canada

    19Institute for Health Policy, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada

    20ICES, Toronto, ON, Canada

    21Department of Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    22Peter Munk Cardiac Centre and the Heart and Stroke Richard Lewar Centre, University of Toronto, Toronto, ON, Canada

    23MAP Centre for Urban Health Solutions, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, ON, Canada

    24College of Pharmacy and Nutrition, University of Saskatchewan, Saskatoon, SK, Canada

    Corresponding Author:

    John L Sievenpiper, MD, PhD

    Department of Nutritional Sciences

    Temerty Faculty of Medicine

    University of Toronto

    5th floor, room 5334, Medical Sciences Building

    1 King's College Circle

    Toronto, ON, M5S 1A8

    Canada

    Phone: 1 416 867 3732

    Fax:1 416 867 7495

    Email: john.sievenpiper@utoronto.ca


    Background: The Portfolio Diet, or Dietary Portfolio, is a therapeutic dietary pattern that combines cholesterol-lowering foods to manage dyslipidemia for the prevention of cardiovascular disease. To translate the Portfolio Diet for primary care, we developed the PortfolioDiet.app as a patient and physician educational and engagement tool for PCs and smartphones. The PortfolioDiet.app is currently being used as an add-on therapy to the standard of care (usual care) for the prevention of cardiovascular disease in primary care. To enhance the adoption of this tool, it is important to ensure that the PortfolioDiet.app meets the needs of its target end users.

    Objective: The main objective of this project is to undertake user testing to inform modifications to the PortfolioDiet.app as part of ongoing engagement in quality improvement (QI).

    Methods: We undertook a 2-phase QI project from February 2021 to September 2021. We recruited users by convenience sampling. Users included patients, family physicians, and dietitians, as well as nutrition and medical students. For both phases, users were asked to use the PortfolioDiet.app daily for 7 days. In phase 1, a mixed-form questionnaire was administered to evaluate the users’ perceived acceptability, knowledge acquisition, and engagement with the PortfolioDiet.app. The questionnaire collected both quantitative and qualitative data, including 2 open-ended questions. The responses were used to inform modifications to the PortfolioDiet.app. In phase 2, the System Usability Scale was used to assess the usability of the updated PortfolioDiet.app, with a score higher than 70 being considered acceptable.

    Results: A total of 30 and 19 users were recruited for phase 1 and phase 2, respectively. In phase 1, the PortfolioDiet.app increased users’ perceived knowledge of the Portfolio Diet and influenced their perceived food choices. Limitations identified by users included challenges navigating to resources and profile settings, limited information on plant sterols, inaccuracies in points, timed-logout frustration, request for step-by-step pop-up windows, and request for a mobile app version; when looking at positive feedback, the recipe section was the most commonly praised feature. Between the project phases, 6 modifications were made to the PortfolioDiet.app to incorporate and address user feedback. At phase 2, the average System Usability Scale score was 85.39 (SD 11.47), with 100 being the best possible.

    Conclusions: By undertaking user testing of the PortfolioDiet.app, its limitations and strengths were able to be identified, informing modifications to the application, which resulted in a clinical tool that better meets users’ needs. The PortfolioDiet.app educates users on the Portfolio Diet and is considered acceptable by users. Although further refinements to the PortfolioDiet.app will continue to be made before its evaluation in a clinical trial, the result of this QI project is an improved clinical tool.

    JMIR Hum Factors 2022;9(2):e34704

    doi:10.2196/34704

    Keywords

    portfolio diet; dietary portfolio; nutrition therapy; dietary application; eHealth; usability testing; quality improvement; mobile phone 


    Background

    The Portfolio Diet, or Dietary Portfolio, is a plant-based dietary pattern of cholesterol-lowering foods that has demonstrated drug-like reductions in low-density lipoprotein cholesterol (LDL-C) and other cardiovascular risk factors [1,2]. In a metabolically controlled study, the Portfolio Diet was shown to result in the same LDL-C reduction (approximately 30%) as lovastatin therapy, the first statin to be widely used [1]. In a recent systematic review and meta-analysis, these drug-like reductions in LDL-C were confirmed and further benefits were also found on other aspects of the lipid profile (non–high-density lipoprotein cholesterol, apolipoprotein B, and triglycerides), blood pressure, inflammatory markers, and estimated 10-year Framingham risk score compared with a National Cholesterol Education Program Step 2 diet alone [3]. This evidence has led to the recognition of the Portfolio Diet as a therapy for cardiovascular disease management from major international clinical practice guidelines, including the Canadian Cardiovascular Society [4,5], Diabetes Canada [6], Obesity Canada [7], Canadian Cardiovascular Harmonized National Guidelines Endeavour [8], Heart UK [9], European Atherosclerosis Society [10], and the American College of Cardiology and American Heart Association guidelines [11]. Although the Portfolio Diet is recognized by clinical practice guidelines as a preventive nutrition therapy for cardiovascular disease, implementation in clinical practice is limited. Traditionally, nutrition therapy involves multiple face-to-face sessions over an extended length of time with trained personnel. However, many health care providers cite a lack of education, educational materials, and time to counsel their patients on nutrition [12,13]. Advancements in technology may be able to circumvent these issues and expand access to nutrition therapies for patients.

    Several studies have shown that health apps can promote positive behavior change and improve related health outcomes. Block et al [14] found that a fully automated intervention targeting nutritional and physical activity behaviors in individuals with prediabetes improved glycemic control and Framingham diabetes risk score over 6 months compared with the waitlist control. In a meta-analysis of 47 randomized controlled trials, Beishuizen et al [15] found that web-based interventions in primary care settings improved risk factors for cardiovascular disease compared with standard of care alone. Thus, health apps can provide an alternative and complementary approach to delivery of preventive nutrition therapy within the limits of primary care, where the shift to remote care during the COVID-19 pandemic has further highlighted the need for evidence-based health apps [16].

    Therefore, to translate the current clinical practice guidelines for nutrition therapy for dyslipidemia, we developed a web-based application, the PortfolioDiet.app. The application was developed by an interdisciplinary team of clinical nutrition experts, registered dietitians, cardiologists, and software architects, as well as patient, physician, and dietitian advisory committees. The collaboration with knowledge users throughout the development and testing process is the central premise of the integrated knowledge translation (iKT) approach [17]. The PortfolioDiet.app is currently being used as an optional add-on therapy to the standard of care (usual care) for primary and secondary prevention of cardiovascular disease at St Michael’s Hospital, Toronto, Ontario, Canada. As part of this iKT approach to enhance the adoption of the Portfolio Diet, it is important to ensure that the PortfolioDiet.app meets the needs of its end users. The population of end users for the application includes adult patients at risk for cardiovascular disease and clinical staff who may wish to learn more about the diet or want to recommend the PortfolioDiet.app to their patients. By undertaking user-centered evaluations, the needs of the target population can be identified, leading to improved uptake of the application.

    Quality Improvement Initiatives

    Quality improvement (QI) initiatives offer an opportunity to optimize and test current clinical tools and are a proven method to improve patient care [18]. These initiatives are especially important when the clinical tool is an app because usability problems have been identified as a major obstacle in the adoption of health apps and have been associated with attrition [19,20]. By performing usability testing of health apps, problems related to ease of use can be identified before undertaking costly trials. Although regarded by many as an essential step in app development, usability testing of nutrition apps is less common in the literature, possibly leading to low user engagement and loss of effectiveness over time [21]. In a recent systematic review by König et al [22], usability was the most frequently identified barrier by participants for nutrition apps, underpinning the importance of usability testing in the development of nutrition therapy apps. This paper provides a description of our user testing approach to help inform research groups seeking to improve similar apps. Therefore, the objective of this project is to undertake and describe user testing to inform modifications to the PortfolioDiet.app as part of ongoing engagement in QI.


    System Intervention

    The PortfolioDiet.app is based on a nutrition therapy to manage dyslipidemia, the Portfolio Diet, that was demonstrated to be effective in individuals with hyperlipidemia [1,2]. The PortfolioDiet.app is a freely available web-based application that can be accessed on any smartphone or PC [23]. A web-based platform was chosen as the initial form to ensure that the application was accessible to patients. Although most Canadians have home internet or smartphone access (94% and 86%, respectively, in 2017) [24], having a web-based platform allows those patients who do not have home internet or a smartphone to access the application through public computers such those as in libraries. Ensuring accessibility was especially important, given the inner city community that St Michael’s Hospital serves. The PortfolioDiet.app is automated and patient facing. The application contains a variety of personalized elements to enhance and sustain patient education and engagement based on a 25-point Portfolio Diet score. These include elements preferred by health app users: an interaction-enabled dashboard, learning resources, gamification components, nudging, and so on [25]. The dashboard presents various summary statistics on adherence, such as total score, individual diet component score, and a 30-day score trend (Multimedia Appendix 1 shows screenshots depicting the various features on the dashboard of the application). The learning resources in the application include the Portfolio Diet infographic, recipes, tip sheets, and educational videos. The infographic provides a visual of the Portfolio Diet and its health benefits (Multimedia Appendix 2). The recipes were developed by registered dietitians according to the Portfolio Diet’s targets. The gamification components include star rewards, weekly quiz questions about the Portfolio Diet, and the Portfolio Diet score leaderboard. Users gain star rewards for each log-in of the day and for completion of weekly quiz questions.

    Design

    We performed a 2-phase QI project from February 2021 to September 2021 (Figure 1). Adult users from a number of areas were invited by email to participate in the testing of the application. Selective convenience sampling was used to generate a varied sample of previously identified end-user groups: patients with hyperlipidemia, family physicians and registered dietitians, the general public, and nutrition and medical students. For both phases, users were provided with the PortfolioDiet.app link and an instructional guide (Multimedia Appendix 3 shows example pages from the instructional guide) and asked to use the PortfolioDiet.app daily for 7 days. As the application is intended to be used over a long-term duration, a 7-day time frame was chosen to ensure that the users had sufficient time to experience each PortfolioDiet.app feature, such as the personalized weekly email reminders and accumulation of their daily scores displayed on the 30-day graph.

    For phase 1, users were sent an email at the end of the 7 days, asking them to complete a mixed-form questionnaire and return it by email (Multimedia Appendix 4). The questionnaire was developed with experts in knowledge uptake evaluation. The purpose of the questionnaire was to evaluate the user’s perceived acceptability, knowledge acquisition, and engagement with the PortfolioDiet.app. The questionnaire collected both quantitative and qualitative data, including 2 open-ended questions. A mixed-form questionnaire allows for a more comprehensive collection of data on views and feedback from end users [26]. Open-ended questions were included to provide users an opportunity to identify strengths and limitations. In phase 2, the usability of the PortfolioDiet.app was measured using the System Usability Scale (SUS; Multimedia Appendix 5). The SUS is a validated usability questionnaire that has been used in clinical settings to assess the usability of various systems and tools [27,28]. The SUS includes 10 statements rated on a 5-point Likert scale. The Likert scale is a psychometric scale often used in psychology questionnaires and frequently applied in health, nutrition, and foods research as well as QI to assess the acceptability of systems and tools. These scales are often used to assess personality, attitudes, and behaviors.

    The application users were asked to indicate their age range (<40 years, 40-60 years, >60 years) because age has been previously identified as an important covariate when assessing usability and is inversely correlated with the SUS score, whereas other characteristics such as gender have not [29]. Other than age range, no other demographic information from users was collected. During the QI project, the core team held weekly meetings to coordinate the application development process. User feedback from phase 1 was discussed within the research team during these weekly meetings. Modifications and updates to address user feedback were implemented in the PortfolioDiet.app and its supporting material before the initiation of phase 2. Although sample sizes of n=5 have previously been deemed acceptable for usability testing [30], a sample of at least 25 was decided upon to ensure a high level of problem detection [31]; therefore, with an anticipated response rate of 80% [32], a total of 30 users were invited. All 30 invitees accepted.

    ‎
    Figure 1. Project overview. SUS: System Usability Scale.
    View this figure

    Data Analysis

    All open-ended responses from the questionnaire were collected and sorted manually into either limitations or strengths. Common comments (reported by ≥2 users) were identified and summarized. Representative quotations of common comments were included to improve the credibility of the findings, an approach recommended by Graneheim and Lundman [33]. All quantitative data were grouped and summarized as totals. A summary usability score was calculated (range 0-100) using the standard score conversion procedure for the SUS [27], with a score higher than 70 being considered acceptable [29]. Data were expressed as mean (SD).

    Ethics Approval

    This project was formally reviewed by institutional authorities at Unity Health Toronto and deemed to require neither research ethics board approval nor written informed consent from participants.


    Phase 1

    For phase 1, a total of 30 users provided feedback from February 2021 to June 2021, with 20 (66%) users aged <40 years, 6 (20%) aged 40-60 years, and 4 (13%) aged >60 years. The response rate for the questionnaire was 100% (30/30). Table S1 in Multimedia Appendix 6 presents the results of the quantitative responses of phase 1. Of the 30 users, 29 (97%) said the PortfolioDiet.app increased their knowledge of the Portfolio Diet, demonstrating that perceived knowledge acquisition was high. Most of the users reported that the application influenced or changed their food choices (24/30, 80%) and that they would use the application daily (20/30, 67%) or weekly (9/30, 30%), demonstrating a high level of engagement with the application. In addition, users ranked the infographic and the tip sheets as the first and second highest features that helped them learn about the Portfolio Diet. Users ranked the star rewards (a gamification component) and recipes as the first and second highest features that supported their interest and engagement in using the application. Most of the users responded that the application was easy to use (26/30, 87%) and it was easy to navigate between the applications functions (28/30, 93%), demonstrating acceptability. Common comments from ≥2 users are summarized as representative quotations in Textbox 1 (a full report of all comments can be found in Table S2 in Multimedia Appendix 6).

    Feedback from phase 1 revealed several opportunities for improvement of the PortfolioDiet.app and its content. The user feedback was reviewed by the team during weekly meetings, and modifications to the second prototype of the application were made to address the common comments. Subsequently, usability of the updated application prototype was assessed in phase 2.

    Qualitative data from users in phase 1.

    Representative quotations of feedback on the PortfolioDiet.app after using it for 7 days.

    1. Limitations and suggestions for improvement
      • “I got more familiar with the food items in each category. I might have learned more but I didn’t realize at first that there was anything important in the ‘Learn’ section.”
      • “I found some portions to be very large.”
      • “...one thing I did not enjoy was the lack of information about plant sterols and where to find/purchase these.”
      • “The app did not accurately record my average scores.”
      • “It would be nice not to have to log in each and every time, if the app could remember my login info.”
      • “It would be helpful to possibly add a video or a step-by-step guide that pops up when you first enter the app. Otherwise, it felt like I had to search for the diet outline and recipes myself.”
      • “Would prefer an actual app, and not doing it via web browser.”
    2. Positives and strengths
      • “I learned a lot about which foods are part of the diet as well as quantities needed for one serving”
      • “The resources gave good summaries of the Portfolio Diet. I was unaware of the Diet prior to beginning using the app so it was a good introduction. The tip sheets and recipes were very helpful.”
      • “I found the front page most useful by allowing me to see where I am not meeting the daily targets, and where and what I still need to eat for the day.”
      • “I loved how easy it was to enter information into the app and the progress bar really helped me visualize my progress.”
      • “The recipe booklet offered many great and creative meal ideas, and I can personally say I have used it since, and will continue to use it moving forward.”
    Textbox 1. Qualitative data from users in phase 1.

    App Updates Based on Themes

    Navigation to Resources: Theme 1

    Users reported navigation challenges with the phase 1 application prototype. For example, some users were unsure how to initially navigate through the PortfolioDiet.app to find resources. Although a PDF instructional guide was provided to all users, this may have not been the most suitable format for communicating with all user types. Therefore, several short videos were created to help supplement the PDF instructional guide, resulting in 9 tutorial videos lasting from 1 minute to 3 minutes to familiarize users with the PortfolioDiet.app and its functions.

    Navigation to Settings: Theme 2

    Multiple users noted concerns with the food portions within the application; a user commented: “I found some portions to be very large.” The application automatically starts all users on the 2000 kcal per day diet. Although instructions for users on changing calorie targets were provided through the PDF instructional guide, it may have not been intuitive to users that the fruit icon at the top of the home page would lead them to their account settings. Therefore, to help users navigate to their account settings, the fruit icon was changed to an Account button (Multimedia Appendix 7 shows the screenshots of the application changes). In addition, a short instructional video was created that explained to users how to correctly select their appropriate kcal target per day in the account settings of the application.

    Plant Sterol Familiarity: Theme 3

    Users expressed limited knowledge of plant sterols. As plant sterols are 1 of 5 major components of the Portfolio Diet, it is important that patients feel informed and comfortable incorporating plant sterols into their diet. It was decided that the creation of an evidence-based educational resource was critical to helping communicate the health benefits of plant sterols to patients and clinical staff. Therefore, a plant sterol tip sheet was developed and added to the updated version of the application (Multimedia Appendix 7).

    Saving Issue Leading to Point Inaccuracy: Theme 4

    Of the 30 users, 2 (7%) expressed concerns regarding inaccuracies in their 25-point Portfolio Diet score calculated by the application. An investigation with the application development team determined that after 45 minutes the application was not connecting with the server and not saving food entries for some users. To address this, a logout notification was added to inform users when to refresh and log back into the application (Multimedia Appendix 7).

    Logout Frustration: Theme 5

    Users expressed frustration with the application automatically logging them out after 45 minutes of inactivity. To help reduce user frustration, the automatic logout was extended to 21 hours as a balance between user experience and personal health data security.

    Opportunities for Future Improvements: Themes 6 and 7

    Although many improvements to the PortfolioDiet.app were made based on phase 1 feedback, certain user feedback remained challenging to address in the short term (Textbox 1: themes 6 and 7). Users suggested the addition of pop-up windows to help with initial navigation to important areas. Although the QI team agreed with the benefits of pop-ups, this proved challenging to implement and was considered lower priority than other key application issues identified by users. In addition, users expressed interest in, or preference for, a mobile app over the current web-based platform. A web-based platform was chosen as the initial form to ensure accessibility of the application. Future work to enhance the adoption of this tool will include the development of an iOS app and an Android app for mobile use as well as the integration of pop-ups to further engage participants with features and resources.

    Recipes: Theme 12

    The enjoyment of the recipes was the most commonly praised feature by users and was the second favorite application feature supporting engagement with the PortfolioDiet.app (Table S1 in Multimedia Appendix 6). Therefore, the recipe bank was expanded from 53 to 70 recipes and culinary students were engaged in this work to expand the cultural diversity of the recipes. To enhance usability, the downloadable PDF recipe book was converted into a filterable recipe webpage, allowing users to filter recipes by each of the Portfolio Diet categories. Recipes were also made filterable by type of meal (eg, breakfast, lunch, dinner, and snack), preparation difficulty level (eg, beginner and intermediate), and preparation time (eg, quick; Multimedia Appendix 7).

    Phase 2

    In phase 2, a total of 19 users completed the SUS from August 2021 to September 2021, with 11 (58%) users aged <40 years, 5 (26%) aged 40-60 years, and 3 (16%) aged >60 years. The response rate was 79% (19/24). Nearly half of the participants (9/19, 47%) were new to using the application. The rest were previously users in the phase 1 testing who were reapproached and asked to again use the updated application for 7 days. The participating users gave the application a mean SUS score of 85.39 (SD 11.47). Full responses to the individual SUS items are shown in Table S3 in Multimedia Appendix 6. Examination of the responses to the individual SUS items showed that most users thought that they would not need the support of a technical person to use the application (average rating of 1.11, SD 0.32, out of 5, where 1=strongly disagree and 5=strongly agree), they thought that the application was easy to use (4.47, SD 0.84), they believed that most people would learn to use the application very quickly (4.68, SD 0.67), and they felt confident using the application (4.32, SD 0.76). There were 2 questions where, although most of the users agreed that they would use the application frequently (3.58, SD 0.90) and that the various functions in the application were well integrated (3.79, SD 1.32), these scores averaged closer to a neutral rating; therefore, updates to further improve application engagement and application function integration will be a focus during the next application revision.


    Principal Findings

    The result of this QI project is a clinical tool that better meets the needs of end users. Through this 2-phase QI project, user feedback was collected and common issues and strengths were identified. The feedback was then used to make modifications to the application. Users considered the updated PortfolioDiet.app as acceptable, giving it a mean SUS score of 85.39 (SD 11.47), which is above the usability quality benchmark threshold score of 70.

    In phase 1, the PortfolioDiet.app was found to increase users’ perceived knowledge of the Portfolio Diet and to influence their perceived food choices. Responses to open-ended questions revealed common issues and suggestions related to challenges with navigating to (1) resources and (2) profile settings, (3) limited information on plant sterols, (4) inaccuracies in points, (5) timed-logout frustration, (6) request for step-by-step pop-up windows, and (7) request for a mobile app version. When looking at positive feedback, the enjoyment of the recipes was the feature most commonly praised by users. Between the project phases, 6 key modifications were made to the PortfolioDiet.app to incorporate user feedback. In phase 2, the participating users gave the updated PortfolioDiet.app a mean SUS score of 85.39 (SD 11.47). The remaining suggestions to be addressed from phase 1 (Textbox 1: themes 6 and 7) should be prioritized in the next update of the application. Moreover, as identified in the SUS findings in phase 2, the focus should be on engagement and function integration to improve the application’s overall usability. To increase engagement, adding social features to the PortfolioDiet.app is recommended. Social features can enhance the benefits of gamification components in engaging users. Patel et al [34] found that a web-based intervention with social support and competition increased physical activity in individuals with type 2 diabetes compared with a control intervention consisting of feedback alone.

    Comparison With Previous Work

    To our knowledge, this is the first QI initiative undertaken with a nutrition therapy application. There is a paucity of literature focused on QI initiatives with health apps in clinical practice. Although trials investigating the benefits of health apps are common, their findings are inconsistent and the details of their QI initiatives are unclear or not reported. The totality of evidence for web-based applications targeting risk factors for cardiovascular disease found beneficial effects on blood pressure, glycated hemoglobin level, LDL-C, body weight, and physical activity compared with standard of care alone in a systematic review and meta-analysis of 47 randomized controlled trials [15]. However, the evidence for the use of mobile apps to improve health outcomes, although positive, was considered weak based on a recent systematic review and meta-analysis [35]. When looking specifically at mobile apps targeting nutrition-related behaviors, Villinger et al [21] found benefits on both nutrition behaviors and nutrition-related health outcomes; however, these benefits were only found in short-term studies lasting for <6 months. The lack of benefits found in longer-term nutrition app studies may be related to low user engagement because of app usability barriers [22]. Nutrition apps are particularly susceptible to usability issues because they require the user to manually enter food data to provide the user with feedback compared with apps that link to accelerometers and other wearable health devices, such as physical activity apps. These inconsistent findings demonstrate the importance of QI and usability testing of health apps before conducting costly trials.

    Previous studies have assessed the usability of digital dietary assessment tools, but these tools were only intended for dietary intake assessment and not for delivering nutrition therapies [36,37]. Usability testing of other lifestyle therapies has been conducted, including a web-based exercise program for older adults (mean SUS score of 84.2, SD 13.3) [38] and a comparison of 2 web-based interventions to increase physical activity, with mean SUS scores of 61.7 (SD 10.8) and 62.5 (SD 11.1) [39]. Another study assessed usability testing of a lifestyle intervention app in patients with type 1 or type 2 diabetes and found a mean SUS score of 62.0 (SD 18.0) [40].

    Strengths and Limitations

    The purpose of this QI project was to conduct initial testing of the PortfolioDiet.app and to integrate a diverse group of end users in the development and testing of the application. By collecting both qualitative and quantitative data, this project allowed for a more comprehensive collection of data on the views of, and feedback from, end users. The data collected through this QI project identified several important issues with the previous application version that were able to be addressed and also provided direction for future development. The expansion of the tailored recipes may be especially important because of concerns in the literature regarding the nutritional content of internet recipes [41]. The COVID-19 pandemic has shown that maintaining care at a distance was not only essential, but must also be done well. The resulting application would strongly support distance care both in times of the pandemic and beyond.

    A limitation of our study was that we used a convenience sample of users. Although convenience samples have been previously found to increase the risk of bias to favor the intervention, we attempted to limit the bias by purposefully reaching out to a broad range of users (patients with hyperlipidemia, family physicians and registered dietitians, the general public, and medical and nutrition students), resulting in an assorted sample. This assorted sample of users may have allowed for more barriers to be identified. The need for more information on plant sterols may have not emerged with a sample of informed patients and staff from a specialist lipid clinic.

    Another limitation is that the SUS was not specifically designed to evaluate therapeutic health apps and is recommended to be combined with other usability metrics. Although there are various methods available to test the usability of therapeutic apps, the SUS is commonly used in the literature [42-44], which allows for comparisons with other therapeutic lifestyle intervention apps. In addition, the SUS uses an intuitive 100-point scale for the score, allowing findings to be easily communicated to those outside of the usability field. Another benefit of the SUS is that it can be completed by users in a short period of time with 10 questions. Other questionnaires developed to assess health apps are longer, increasing user response burden. Future assessments of the mobile version of the PortfolioDiet.app will include other questionnaires more specific to mobile health apps, such as the user version of the Mobile Application Rating Scale questionnaire [45].

    Conclusions

    The continued consultation with knowledge users throughout the development and testing process of the PortfolioDiet.app aligns with participatory research or iKT approaches [17]. The result of this QI project is a clinical tool that better meets the needs of end users. Although the therapeutic benefits of the Portfolio Diet are well established and the PortfolioDiet.app was demonstrated to increase knowledge of the Portfolio Diet and is usable, the impact of the PortfolioDiet.app on LDL-C and cardiovascular risk reduction is unknown. Therefore, the next step will be to evaluate the utility of the PorfolioDiet.app in primary care settings in a clinical trial.

    Acknowledgments

    The authors would like to thank all participating users for their time and detailed feedback. The authors would also like to thank their PortfolioDiet.app team and the many volunteers who assisted in the development and formatting of recipes and other important resources used in the application [46]. The Toronto 3D Knowledge Synthesis and Clinical Trials foundation provided funding for the project. The Diet, Digestive tract, and Disease (3D) Centre, funded through the Canada Foundation for Innovation and the Ministry of Research and Innovation’s Ontario Research Fund, provided the infrastructure for the conduct of this work. MEK was funded by a Toronto 3D Knowledge Synthesis and Clinical Trials foundation PhD Scholarship Award. LC was funded by a Mitacs-Elevate Postdoctoral Fellowship Award. AJG was supported by a Canadian Institute of Health Research Fellowship. GH was funded by a Toronto 3D Summer Student Scholarship. DJAJ was funded by the Government of Canada through the Canada Research Chair Endowment. JLS was funded by a PSI Graham Farquharson Knowledge Translation Fellowship; Canadian Diabetes Association Clinician Scientist Award; Canadian Institute of Health Research INMD and CNS New Investigator Partnership Prize; and Banting & Best Diabetes Centre Sun Life Financial New Investigator Award.

    Authors' Contributions

    All authors were involved in the development and testing of the PortfolioDiet.app prototypes. MEK, LC, AJG, GH, CMC, and JLS met weekly to discuss feedback and prototype updates. MEK contributed to the project design and questionnaire development, analyzed and interpreted the data, wrote the first draft of the manuscript, incorporated revisions, and approved the final version. LC acquired the data, contributed to the project design and questionnaire development, analyzed and interpreted the data, revised the manuscript for important intellectual content, and approved the final version. AJG contributed to the project design, analyzed and interpreted the data, revised the manuscript for important intellectual content, and approved the final version. GH assisted with acquiring the data, contributed to the project design, analyzed and interpreted the data, revised the manuscript for important intellectual content, and approved the final version. CMC developed the different prototypes, revised the manuscript for important intellectual content, and approved the final version. SMG led questionnaire and methods development, revised the manuscript for important intellectual content, and approved the final version. WW, AL, CH, JR, and KS were part of the physician advisory committee and were involved in the design of the application. DS provided expertise in behavior change theory. RGJ, VSM, WW, AL, CH, JR, KS, DS, ESC, JAU, PJ, GLB, MEF, LAL, CWCK, and DJAJ provided input into the project and application design, revised the manuscript for important intellectual content, and approved the final version. JLS provided project supervision, revised the manuscript for important intellectual content, and approved the final version.

    Conflicts of Interest

    MEK has received funding from a Toronto 3D Knowledge Synthesis and Clinical Trials foundation PhD scholarship award and is a part-time employee at Inquis Clinical Research, a contract research organization. LC was a Mitacs Elevate postdoctoral fellow jointly funded by the government of Canada and Canadian Sugar Institute (from February 2019 to August 2021). AJG has received consulting fees from SoLo GI Nutrition and an honorarium from the Soy Nutrition Institute. PJ serves as an unpaid member of the steering group of trials funded by Appili Therapeutics (Control COVID-19 trial: ClinicalTrials.gov NCT04448119), Abbott Vascular (Evaluation of XIENCE Versus Coronary Artery Bypass Surgery for Effectiveness of Left Main Revascularization [EXCEL] trial: ClinicalTrials.gov NCT01205776; comparing XIENCE stent in participants with unprotected left main coronary artery disease with coronary artery bypass graft surgery; no active involvement for >3 years, no coauthored publication, but still listed as an original member of the statistical executive committee), and Terumo (Management of High Bleeding Risk Patients Post Bioresorbable Polymer Coated Stent Implantation With an Abbreviated Versus Standard DAPT [dual antiplatelet therapy] Regimen [MASTER DAPT] trial: ClinicalTrials.gov NCT03023020; comparing abbreviated DAPT with prolonged DAPT in patients with a drug-eluting stent; ongoing active involvement as a member of the steering group). MEF has received research support from Amgen; Astra Zeneca; Novartis; and Novo Nordisk. CWCK has received grants or research support from the Advanced Foods and Materials Network, Agriculture and Agri-Food Canada, Almond Board of California, American Pistachio Growers, Barilla, Calorie Control Council, Canadian Institutes of Health Research, Canola Council of Canada, International Nut and Dried Fruit Council, International Tree Nut Council Nutrition Research & Education Foundation, Loblaw Companies Ltd, Pulse Canada, Saskatchewan Pulse Growers Association, and Unilever. He has received in-kind research support from the Almond Board of California, American Peanut Council, Barilla, California Walnut Commission, Kellogg Canada, Loblaw Companies Ltd, Quaker (PepsiCo), Primo, Unico, Unilever, and WhiteWave Foods; has received travel support or honoraria from the American Peanut Council, American Pistachio Growers, Barilla, California Walnut Commission, Canola Council of Canada, General Mills, International Nut and Dried Fruit Council, International Pasta Organization, Loblaw Companies Ltd, Nutrition Foundation of Italy, Oldways Preservation Trust, Paramount Farms, Peanut Institute, Pulse Canada, Sabra Dipping, Saskatchewan Pulse Growers Association, Sun-Maid, Tate & Lyle, Unilever, and WhiteWave Foods. He has served on the scientific advisory board for the International Tree Nut Council, International Pasta Organization, Lantmannen, McCormick Science Institute, Oldways Preservation Trust, Paramount Farms, and Pulse Canada. He is a member of the International Carbohydrate Quality Consortium, executive board member of the diabetes and nutrition study group of the European Association for the Study of Diabetes; is on the clinical practice guidelines expert committee for nutrition therapy of the European Association for the Study of Diabetes; and is a director of the Toronto 3D Knowledge Synthesis and Clinical Trials foundation. DJAJ has received research grants from the Saskatchewan and Alberta Pulse Growers Associations; the Agricultural Bioproducts Innovation Program through the Pulse Research Network; Advanced Foods and Materials Network; Loblaw Companies Ltd; Unilever Canada and Netherlands; Barilla; Almond Board of California; Agriculture and Agri-Food Canada; Pulse Canada; Kellogg Canada; Quaker Oats Canada; Procter & Gamble Technical Centre Ltd; Bayer Consumer Care, Springfield, New Jersey; Quaker (PepsiCo); International Nut and Dried Fruit Council; Soy Foods Association of North America; the Coca-Cola Company (investigator-initiated, unrestricted grant); Solae; Haine Celestial; the Sanitarium Company; Orafti; International Tree Nut Council Nutrition Research & Education Foundation; Peanut Institute; Soy Nutrition Institute; Canola and Flax Councils of Canada; Calorie Control Council; Canadian Institutes of Health Research; Canada Foundation for Innovation; and the Ontario Research Fund. He has received in-kind supplies for trials as research support from the Almond Board of California, Walnut Council of California, Peanut Institute, Barilla, Unilever, Unico, Primo, Loblaw Companies Ltd, Quaker (PepsiCo), Pristine Gourmet, Bunge Limited, Kellogg Canada, and WhiteWave Foods. He has been on the speakers’ panel, served on the scientific advisory board, and received travel support and honoraria from Nutritional Fundamentals for Health–Nutramedica; Saint Barnabas Medical Center; the University of Chicago; 2020 China Glycemic Index International Conference; Atlantic Pain Conference; Academy of Life Long Learning; Almond Board of California; Canadian Agriculture Policy Institute; Loblaw Companies Ltd; Griffin Hospital (for the development of the NuVal scoring system); the Coca-Cola Company; Epicure; Danone; Diet Quality Photo Navigation; Better Therapeutics (formerly known as FareWell); Verywell; True Health Initiative; Heali AI Corp; Institute of Food Technologists; Soy Nutrition Institute; Herbalife Nutrition Institute; the Saskatchewan and Alberta Pulse Growers Associations; Sanitarium Company; Orafti; International Tree Nut Council Nutrition Research & Education Foundation; Peanut Institute; Herbalife International; Pacific Health Laboratories; Barilla; Metagenics; Bayer Consumer Care; Unilever Canada and Netherlands; Solae; Kellogg; Quaker Oats; Procter & Gamble; Abbott Laboratories; Dean Foods; California Strawberry Commission; Haine Celestial; PepsiCo; Alpro Foundation; Pioneer Hi-Bred International; DuPont Nutrition and Health; Spherix Consulting; WhiteWave Foods; Advanced Foods and Materials Network; Canola and Flax Councils of Canada; Agriculture and Agri-Food Canada; Canadian Agri-Food Policy Institute; Pulse Canada; Soy Foods Association of North America; Nutrition Foundation of Italy; Nutra-Source Diagnostics; the McDougall Program; Toronto Knowledge Translation Group (St Michael’s Hospital); Canadian College of Naturopathic Medicine; the Hospital for Sick Children; Canadian Nutrition Society; American Society for Nutrition; Arizona State University; Paolo Sorbini Foundation; and Institute of Nutrition, Metabolism and Diabetes. He received an honorarium from the US Department of Agriculture to present the 2013 WO Atwater Memorial Lecture. He received the 2013 Award for Excellence in Research from the International Nut and Dried Fruit Council. He received funding and travel support from the Canadian Society of Endocrinology and Metabolism to produce mini-cases for the Canadian Diabetes Association. He is a member of the International Carbohydrate Quality Consortium. His wife, Alexandra L Jenkins, is a director and partner of Inquis Clinical Research for the food industry; his two daughters, Wendy Jenkins and Amy Jenkins, have published a book for vegetarians that promotes the use of the foods described here, The Portfolio Diet for Cardiovascular Risk Reduction (Academic Press [Elsevier] 2020, ISBN: 978-0-12-810510-8); and his sister, Caroline Brydson, received funding through a grant from the St Michael’s Hospital Foundation to develop a cookbook for one of his studies. He is also a vegan. JLS has received research support from the Canadian Foundation for Innovation; Ontario Research Fund; Province of Ontario Ministry of Research, Innovation and Science; Canadian Institutes of Health Research; Diabetes Canada; PSI Foundation; Banting & Best Diabetes Centre; American Society for Nutrition; International Nut and Dried Fruit Council Foundation; National Dried Fruit Trade Association; National Honey Board (the US Department of Agriculture honey Checkoff program); International Life Sciences Institute; Pulse Canada; Quaker Oats Center of Excellence; United Soybean Board (the US Department of Agriculture soy Checkoff program); the Tate & Lyle Nutritional Research Fund at the University of Toronto; the Glycemic Control and Cardiovascular Disease in Type 2 Diabetes Fund at the University of Toronto (a fund established by the Alberta Pulse Growers Association); and the Nutrition Trialists Fund at the University of Toronto (a fund established by an inaugural donation from the Calorie Control Council). He has received in-kind food donations to support a randomized controlled trial from the Almond Board of California, California Walnut Commission, Peanut Institute, Barilla, Upfield, Unilever, Unico, Primo, Loblaw Companies Ltd, Quaker, Kellogg Canada, WhiteWave Foods (Danone), and Nutrartis. He has received travel support, speaker fees, and honoraria from Diabetes Canada, Dairy Farmers of Canada, FoodMinds LLC, International Sweeteners Association, Nestlé, Pulse Canada, Canadian Society for Endocrinology and Metabolism, GI Foundation, Abbott, General Mills, Biofortis, the American Society for Nutrition, Northern Ontario School of Medicine, International Tree Nut Council Nutrition Research & Education Foundation, European Food Safety Authority, Comité Européen des Fabricants de Sucre, Nutrition Communications, International Food Information Council, Calorie Control Council, and Physicians Committee for Responsible Medicine. He has, or has had, ad hoc consulting arrangements with Perkins Coie LLP, Tate & Lyle, Wirtschaftliche Vereinigung Zucker eV, Danone, and Inquis Clinical Research. He is a member of the European Fruit Juice Association Scientific Expert Panel and a former member of the Soy Nutrition Institute scientific advisory committee. He is on the clinical practice guidelines expert committees of Diabetes Canada, European Association for the Study of Diabetes, and Canadian Cardiovascular Society, as well as Obesity Canada and the Canadian Association of Bariatric Physicians and Surgeons. He serves, or has served, as an unpaid scientific advisor for the food, nutrition, and safety program and the technical committee on carbohydrates of International Life Sciences Institute North America. He is a member of the International Carbohydrate Quality Consortium, executive board member of the diabetes and nutrition study group of the European Association for the Study of Diabetes, and director of the Toronto 3D Knowledge Synthesis and Clinical Trials foundation. His wife is an employee of AB InBev.

    Authors GH, CMC, SMG, RGJ, VSM, WW, AL, CH, JR, KS, DS, ESC, JAU, GLB, LAL have no conflicts of interest to disclose.

    ‎
    Multimedia Appendix 1

    Screenshots depicting the various features on the dashboard of the application.

    PDF File (Adobe PDF File), 760 KB
    ‎
    Multimedia Appendix 2

    The Portfolio Diet infographic.

    PDF File (Adobe PDF File), 2586 KB
    ‎
    Multimedia Appendix 3

    Example pages from the instructional guide (navigation and progress).

    PDF File (Adobe PDF File), 363 KB
    ‎
    Multimedia Appendix 4

    Mixed-form feedback questionnaire (phase 1).

    DOCX File , 20 KB
    ‎
    Multimedia Appendix 5

    System Usability Scale (phase 2).

    DOCX File , 19 KB
    ‎
    Multimedia Appendix 6

    Tables showing full quantitative responses (phase 1), full qualitative responses broken up by limitations and strengths (phase 1), and scores for individual System Usability Scale items (phase 2).

    DOCX File , 36 KB
    ‎
    Multimedia Appendix 7

    Screenshots depicting updates made to the application based on user feedback.

    PDF File (Adobe PDF File), 2420 KB
    1. Jenkins DJ, Kendall CW, Marchie A, Faulkner DA, Wong JM, de Souza R, et al. Effects of a dietary portfolio of cholesterol-lowering foods vs lovastatin on serum lipids and C-reactive protein. JAMA 2003;290(4):502-510. [CrossRef] [Medline]
    2. Jenkins DJ, Jones PJ, Lamarche B, Kendall CW, Faulkner D, Cermakova L, et al. Effect of a dietary portfolio of cholesterol-lowering foods given at 2 levels of intensity of dietary advice on serum lipids in hyperlipidemia: a randomized controlled trial. JAMA 2011;306(8):831-839. [CrossRef] [Medline]
    3. Chiavaroli L, Nishi SK, Khan TA, Braunstein CR, Glenn AJ, Mejia SB, et al. Portfolio dietary pattern and cardiovascular disease: a systematic review and meta-analysis of controlled trials. Prog Cardiovasc Dis 2018;61(1):43-53 [FREE Full text] [CrossRef] [Medline]
    4. Pearson G, Thanassoulis G, Anderson T, Barry AR, Couture P, Dayan N, et al. 2021 Canadian cardiovascular society guidelines for the management of dyslipidemia for the prevention of cardiovascular disease in adults. Can J Cardiol 2021;37(8):1129-1150. [CrossRef] [Medline]
    5. Anderson TJ, Grégoire J, Pearson GJ, Barry AR, Couture P, Dawes M, et al. 2016 Canadian cardiovascular society guidelines for the management of dyslipidemia for the prevention of cardiovascular disease in the adult. Can J Cardiol 2016;32(11):1263-1282. [CrossRef] [Medline]
    6. Diabetes Canada Clinical Practice Guidelines Expert Committee, Punthakee Z, Goldenberg R, Katz P. Definition, classification and diagnosis of diabetes, prediabetes and metabolic syndrome. Can J Diabetes 2018;42 Suppl 1:S10-S15. [CrossRef] [Medline]
    7. Brown J, Clarke C, Johnson Stoklossa V, Sievenpiper J. Canadian adult obesity clinical practice guidelines: medical nutrition therapy in obesity management. Obesity Canada. 2020.   URL: https://obesitycanada.ca/guidelines/nutrition [accessed 2021-03-25]
    8. Tobe SW, Stone JA, Anderson T, Bacon S, Cheng AY, Daskalopoulou SS, et al. Canadian cardiovascular harmonized national guidelines endeavour (C-CHANGE) guideline for the prevention and management of cardiovascular disease in primary care: 2018 update. CMAJ 2018;190(40):E1192-E1206 [FREE Full text] [CrossRef] [Medline]
    9. Portfolio diet. Heart UK: The cholesterol charity.   URL: https://www.heartuk.org.uk/downloads/healthprofessionals/factsheets/the-portfolio-diet.pdf [accessed 2016-02-04]
    10. Stroes ES, Thompson PD, Corsini A, Vladutiu GD, Raal FJ, Ray KK, European Atherosclerosis Society Consensus Panel. Statin-associated muscle symptoms: impact on statin therapy-European atherosclerosis society consensus panel statement on assessment, aetiology and management. Eur Heart J 2015;36(17):1012-1022 [FREE Full text] [CrossRef] [Medline]
    11. Grundy SM, Stone NJ, Bailey AL, Beam C, Birtcher KK, Blumenthal RS, et al. 2018 AHA/ACC/AACVPR/AAPA/ABC/ACPM/ADA/AGS/APhA/ASPC/NLA/PCNA guideline on the management of blood cholesterol: executive summary: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. J Am Coll Cardiol 2019;73(24):3168-3209. [CrossRef] [Medline]
    12. Kris-Etherton PM, Akabas SR, Bales CW, Bistrian B, Braun L, Edwards MS, et al. The need to advance nutrition education in the training of health care professionals and recommended research to evaluate implementation and effectiveness. Am J Clin Nutr 2014;99(5 Suppl):1153S-1166S [FREE Full text] [CrossRef] [Medline]
    13. Aboueid S, Bourgeault I, Giroux I. Nutrition and obesity care in multidisciplinary primary care settings in Ontario, Canada: short duration of visits and complex health problems perceived as barriers. Prev Med Rep 2018;10:242-247 [FREE Full text] [CrossRef] [Medline]
    14. Block G, Azar KM, Romanelli RJ, Block TJ, Hopkins D, Carpenter HA, et al. Diabetes prevention and weight loss with a fully automated behavioral intervention by email, web, and mobile phone: a randomized controlled trial among persons with prediabetes. J Med Internet Res 2015;17(10):e240 [FREE Full text] [CrossRef] [Medline]
    15. Beishuizen CR, Stephan BC, van Gool WA, Brayne C, Peters RJ, Andrieu S, et al. Web-based interventions targeting cardiovascular risk factors in middle-aged and older people: a systematic review and meta-analysis. J Med Internet Res 2016;18(3):e55 [FREE Full text] [CrossRef] [Medline]
    16. Bruining N. The post-pandemic legacy: the breakthrough of digital health and telemedicine. Cardiovasc Res 2021;117(9):e118-e119 [FREE Full text] [CrossRef] [Medline]
    17. Gagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci 2016;11(1):38 [FREE Full text] [CrossRef] [Medline]
    18. Quality improvement guide. Health Quality Ontario (HQO). 2012.   URL: https://www.hqontario.ca/portals/0/documents/qi/qi-quality-improve-guide-2012-en.pdf [accessed 2021-09-13]
    19. Brown 3rd W, Yen PY, Rojas M, Schnall R. Assessment of the health IT usability evaluation model (Health-ITUEM) for evaluating mobile health (mHealth) technology. J Biomed Inform 2013;46(6):1080-1087 [FREE Full text] [CrossRef] [Medline]
    20. Eysenbach G. The law of attrition. J Med Internet Res 2005;7(1):e11 [FREE Full text] [CrossRef] [Medline]
    21. Villinger K, Wahl DR, Boeing H, Schupp HT, Renner B. The effectiveness of app-based mobile interventions on nutrition behaviours and nutrition-related health outcomes: a systematic review and meta-analysis. Obes Rev 2019;20(10):1465-1484 [FREE Full text] [CrossRef] [Medline]
    22. König LM, Attig C, Franke T, Renner B. Barriers to and facilitators for using nutrition apps: systematic review and conceptual framework. JMIR Mhealth Uhealth 2021;9(6):e20037 [FREE Full text] [CrossRef] [Medline]
    23. The Portfolio Diet. 2021.   URL: https://portfoliodiet.app/ [accessed 2021-11-03]
    24. Paré G, Bourget C. Diffusion of smart devices for health in Canada. Canada Health Infoway. 2017.   URL: https://tinyurl.com/bdd3n3en [accessed 2021-10-07]
    25. Coorey GM, Neubeck L, Mulley J, Redfern J. Effectiveness, acceptability and usefulness of mobile applications for cardiovascular disease self-management: systematic review with meta-synthesis of quantitative and qualitative data. Eur J Prev Cardiol 2018;25(5):505-521. [CrossRef] [Medline]
    26. Curry LA, Krumholz HM, O'Cathain A, Plano Clark VL, Cherlin E, Bradley EH. Mixed methods in biomedical and health services research. Circ Cardiovasc Qual Outcomes 2013;6(1):119-123. [CrossRef] [Medline]
    27. Brooke J. SUS: a 'quick and dirty' usability scale. In: Jordan PW, Thomas B, McClelland I, Weerdmeester B, editors. Usability evaluation in industry. Boca Raton, FL: CRC Press; 1996.
    28. Lewis JR, Sauro J. The factor structure of the system usability scale. In: International Conference on Human Centered Design. 2009 Presented at: HCD '09; July 19-24, 2009; San Diego, CA p. 94-103. [CrossRef]
    29. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact 2008;24(6):574-594. [CrossRef]
    30. Nielsen J. Why you only need to test with 5 users. Nielsen Norman Group. 2000.   URL: https://www.nngroup.com/articles/how-many-test-users/ [accessed 2021-10-13]
    31. Macefield R. How to specify the participant group size for usability studies: a practitioner's guide. J Usability Stud 2009;5(1):34-45.
    32. Melin J, Bonn SE, Pendrill L, Trolle Lagerros Y. A questionnaire for assessing user satisfaction with mobile health apps: development using Rasch measurement theory. JMIR Mhealth Uhealth 2020;8(5):e15909 [FREE Full text] [CrossRef] [Medline]
    33. Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today 2004;24(2):105-112. [CrossRef] [Medline]
    34. Patel MS, Small DS, Harrison JD, Hilbert V, Fortunato MP, Oon AL, et al. Effect of behaviorally designed gamification with social incentives on lifestyle modification among adults with uncontrolled diabetes: a randomized clinical trial. JAMA Netw Open 2021;4(5):e2110255 [FREE Full text] [CrossRef] [Medline]
    35. Iribarren SJ, Akande TO, Kamp KJ, Barry D, Kader YG, Suelzer E. Effectiveness of mobile apps to promote health and manage disease: systematic review and meta-analysis of randomized controlled trials. JMIR Mhealth Uhealth 2021;9(1):e21563 [FREE Full text] [CrossRef] [Medline]
    36. Albar SA, Carter MC, Alwan NA, Evans CE, Cade JE. Formative evaluation of the usability and acceptability of myfood24 among adolescents: a UK online dietary assessments tool. BMC Nutr 2015;1(1):1. [CrossRef]
    37. Zenun Franco R, Fallaize R, Lovegrove JA, Hwang F. Online dietary intake assessment using a graphical food frequency app (eNutri): usability metrics from the EatWellUK study. PLoS One 2018;13(8):e0202006 [FREE Full text] [CrossRef] [Medline]
    38. Dekker-van Weering M, Jansen-Kosterink S, Frazer S, Vollenbroek-Hutten M. User experience, actual use, and effectiveness of an information communication technology-supported home exercise program for pre-frail older adults. Front Med (Lausanne) 2017;4:208 [FREE Full text] [CrossRef] [Medline]
    39. Vandelanotte C, Kolt GS, Caperchione CM, Savage TN, Rosenkranz RR, Maeder AJ, et al. Effectiveness of a web 2.0 intervention to increase physical activity in real-world settings: randomized ecological trial. J Med Internet Res 2017;19(11):e390 [FREE Full text] [CrossRef] [Medline]
    40. Fu HN, Adam TJ, Konstan JA, Wolfson JA, Clancy TR, Wyman JF. Influence of patient characteristics and psychological needs on diabetes mobile app usability in adults with type 1 or type 2 diabetes: crossover randomized trial. JMIR Diabetes 2019;4(2):e11462 [FREE Full text] [CrossRef] [Medline]
    41. Trattner C, Elsweiler D, Howard S. Estimating the healthiness of internet recipes: a cross-sectional study. Front Public Health 2017;5:16 [FREE Full text] [CrossRef] [Medline]
    42. Edney SM, Olds TS, Ryan JC, Vandelanotte C, Plotnikoff RC, Curtis RG, et al. A social networking and gamified app to increase physical activity: cluster RCT. Am J Prev Med 2020;58(2):e51-e62. [CrossRef] [Medline]
    43. Ormel HL, van der Schoot GG, Westerink NL, Sluiter WJ, Gietema JA, Walenkamp AM. Self-monitoring physical activity with a smartphone application in cancer patients: a randomized feasibility study (SMART-trial). Support Care Cancer 2018;26(11):3915-3923 [FREE Full text] [CrossRef] [Medline]
    44. Rayward AT, Murawski B, Duncan MJ, Holliday EG, Vandelanotte C, Brown WJ, et al. Efficacy of an m-health physical activity and sleep intervention to improve sleep quality in middle-aged adults: the refresh study randomized controlled trial. Ann Behav Med 2020;54(7):470-483. [CrossRef] [Medline]
    45. Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and validation of the user version of the mobile application rating scale (uMARS). JMIR Mhealth Uhealth 2016;4(2):e72 [FREE Full text] [CrossRef] [Medline]
    46. Our team. The Portfolio Diet App. 2021.   URL: https://recipes.portfoliodiet.app/our-team/ [accessed 2021-09-10]

    ‎
    iKT: integrated knowledge translation
    LDL-C: low-density lipoprotein cholesterol
    QI: quality improvement
    SUS: System Usability Scale

    Edited by A Kushniruk; submitted 03.12.21; peer-reviewed by G Block, S Jansen-Kosterink; comments to author 29.12.21; revised version received 04.02.22; accepted 13.02.22; published 21.04.22

    Copyright

    ©Meaghan E Kavanagh, Laura Chiavaroli, Andrea J Glenn, Genevieve Heijmans, Shannan M Grant, Chi-Ming Chow, Robert G Josse, Vasanti S Malik, William Watson, Aisha Lofters, Candice Holmes, Julia Rackal, Kristie Srichaikul, Diana Sherifali, Erna Snelgrove-Clarke, Jacob A Udell, Peter Juni, Gillian L Booth, Michael E Farkouh, Lawrence A Leiter, Cyril WC Kendall, David JA Jenkins, John L Sievenpiper. Originally published in JMIR Human Factors (https://humanfactors.jmir.org), 21.04.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on https://humanfactors.jmir.org, as well as this copyright and license information must be included.