Published on in Vol 5, No 2 (2018): Apr-Jun

Preprints (earlier versions) of this paper are available at, first published .
Relationship Between Evidence Requirements, User Expectations, and Actual Experiences: Usability Evaluation of the Twazon Arabic Weight Loss App

Relationship Between Evidence Requirements, User Expectations, and Actual Experiences: Usability Evaluation of the Twazon Arabic Weight Loss App

Relationship Between Evidence Requirements, User Expectations, and Actual Experiences: Usability Evaluation of the Twazon Arabic Weight Loss App

Original Paper

1Food Science and Nutrition Department, College of Food and Agriculture Sciences, King Saud University, Riyadh, Saudi Arabia

2Institute of Applied Health Sciences, University of Aberdeen, Aberdeen, United Kingdom

3Warwick Medical School, University of Warwick, Coventry, United Kingdom

Corresponding Author:

Aroub Alnasser, BSc (Hons), MSc

Food Science and Nutrition Department

College of Food and Agriculture Sciences

King Saud University

PO Box 86683

Riyadh, 11632

Saudi Arabia

Phone: 966 118056476


Background: Saudi Arabia has faced a steady growth in the prevalence of obesity. The concurrent and ubiquitous use of mobile technology, such as smartphones and apps, provides an opportunity for the implementation of mHealth technology, a method for delivering behavioral interventions. Despite their effectiveness in promoting lifestyle and diet modification, culturally adapted weight loss apps and related interventions are lacking in Gulf Cooperation Council countries.

Objective: The objective of our study was to identify the relationship between adherence to evidence-informed practices, potential user expectations, and actual user experiences in order to enhance the understanding of the overall usability of the Twazon Arabic weight loss app.

Methods: In 2 previous studies, 39 Saudi women were recruited for focus group discussions and 240 Saudi women were recruited for an app-based weight loss intervention. Usability of the Twazon Arabic weight loss app was evaluated by analyzing the opinions and experiences of 26 participants who engaged with the Twazon app for 4 months; the System Usability Scale (SUS) and word clouds were used. The results were triangulated with potential user expectations obtained in the focus group discussion and with the findings from an Arabic app screening for evidence-informed practices.

Results: The average reported SUS score was 69.3. The most favored features were the calorie counter, step counter, and physical activity calorie counter. The features in need of improvement were the social network, notifications, and the Twazon Saudi Food Database. Twazon users preferred and found useful 7 of the 13 evidence-informed weight loss practices that were integrated into the features of the app.

Conclusions: Triangulation identified the most notable relationship to be the disparity between user experience and 2 of the evidence-informed practices, namely a minimum weight loss goal of 0.5 to 1 kg/week and social support; no relationship was found between user expectations and evidence-informed weight loss practices. The overall usability of the Twazon Arabic weight loss app ranged between high marginal and acceptable, indicating that some improvements to the app should be considered for implementation in future app-based weight loss interventions of this kind.

JMIR Hum Factors 2018;5(2):e16



It is no longer news that obesity is a problem in Gulf Cooperation Council countries such as Saudi Arabia, affecting more women than men on average. A major driver of this is unhealthy behaviors such as physical inactivity, overeating, and unhealthful food choices [1]. Due to the severity of the epidemic, it is necessary to implement various treatment strategies and conduct interventions that are accessible to a larger population and effective over the long term. As a novel manner in which to deliver behavioral interventions that might be effective in lifestyle and diet modification, implementation of health-related technology, or rather mHealth, has been of emerging interest.

mHealth is a type of electronic health support that is defined as medical and public health practices that are promoted by mobile devices, such as smartphones, patient monitoring devices, personal digital assistants, and other wireless devices [2]. Commercial weight loss apps have been reported to be more engaging than those that are evidence-informed [3]; however, the quality of the information given by the commercial apps is often rated as low [4,5]. It follows that a more comprehensive user-centered design approach [6] that is based on evidence-informed practices, as well as user expectations and experiences, is vital to ensuring the efficacy of mHealth interventions.

Due to the widespread use and accessibility of mobile technology in Saudi Arabia [7], smartphone apps offer a substantial opportunity to support health behavior change and weight management. However, none that are evidence-informed and culturally adapted are available in the region. With the goal of implementing a 4-month weight loss intervention in Saudi Arabia (AA et al, unpublished data, 2017), the Twazon Arabic weight loss app [8] was developed based on the aforementioned factors in addition to behavior strategies [9], such as self-monitoring. To ensure the proper implementation of a complex intervention [10] involving a website or mobile app, usability —or how effectively, efficiently, and satisfactorily a user can interact with a user interface [11]—must be investigated.

The Twazon app was designed to be used autonomously by individuals (male or female) who have weight issues, but are otherwise healthy; it is not intended to be used as treatment in a health care system. The prevalence of overweight and obesity among Saudi women, and a scarcity of research done for this demographic, justify the need for a public health intervention to be carried out for women in this region. In this study, we aimed to identify the relationship between adherence to evidence-informed practices, potential user expectations, and actual user experiences in order to enhance the understanding of the overall usability of the Twazon weight loss app. A triangulation analysis revealed the relationship and tensions found between these aspects of the app’s components, and the results we report here reflect their compliance with the Twazon app.

Design Phase: Evidence Requirements and User Expectations

Weight loss apps in general have been found to be lacking in evidence-informed practices, and the majority that are available are commercial and in English. Due to a complete lack of a systematic reviews of weight loss apps in the region, the first step in designing Twazon [8] involved screening 65 Arabic weight loss apps for their adherence to evidence-informed practices [12] as recommended by various health authorities [13-15].

To further inform the development of the Twazon app, a qualitative study was conducted comprising 4 focus group discussions with the goal of determining potential users’ preferences and expectations in a weight loss app. A total of 39 Saudi women with overweight and obesity in Riyadh, Saudi Arabia [16] gave oral responses, which were transcribed and translated from Arabic into English by a certified bilingual translator. Discussions were thematically analyzed and categorized for each of the main topics, and specific quotations were identified to correlate with the theme in mind.

Implementation Phase: App Development and Intervention

The result of the app screening and the focus group discussions was the selection of 13 individual evidence-informed practices, which were grouped as follows: weight assessment and goal setting, healthy diet, physical activity, self-monitoring, and social support (see Multimedia Appendix 1). The behavioral strategy of self-monitoring translated to features of the Twazon app that enable users to track their progress.

The Twazon app requires a single log-in, must be connected to the Internet to function properly, and continues to work in the background. It is not a commercial app; it was developed and made freely available to the public through the iTunes (Apple Inc) and Google Play (Google LLC) stores. Daily physical activity, by activity and time spent doing it, is calculated with data from the user-updated physical activity journal and the integrated pedometer; daily water and energy intake are calculated based on user-updated input of consumption (Figure 1 shows the Twazon app interface).

The dashboard provides automatized, individually tailored, user-specific information regarding daily activity, consumption, and goal tracking, which is reset at the beginning of each day by an automatic algorithm. The food palm gives a personalized biweekly graphic display of the user’s healthy lifestyle self-assessment score, including physical activity tips when physical activity goals are not met. This feature is also designed to give instant feedback to users if they exceed their daily energy intake goal. The educational tool is used for menu planning, and the food label tips are used to understand the nutritional content of foods consumed. The Twazon app also offers social support, accessible at the bottom of the interface. This social network, which is restricted to users, encourages individuals to share personal health achievements with one another through the posting and liking of images and text; no other human contact or feedback from the developers is provided.

Postintervention Phase: App Use and Usability

Engagement was based on app use, which was calculated by an automatic algorithm that grouped the participants according to the frequency of user input (AA et al, unpublished data, 2017). This was a necessary step in assessing usability in that only those participants who regularly updated their information could be considered.

Generally, usability testing conducted with 5 participants will identify at least 85% of usability problems [17]; in this study, a sample of 26 users was deemed to be more than sufficient. Participants were asked to assess the overall usability of the app during an individual interview at a predetermined location for 10 to 15 minutes. The overall usability score of the Twazon app was measured using the 10-question System Usability Scale (SUS), which generates a SUS score ranging from 0 to 100 that is associated with a 7-point adjective rating scale: worst imaginable (12.5), awful (20.3), poor (35.7), ok (50.9), good (71.4), excellent (85.5), or best imaginable (90.9) [18]. To determine what is an “acceptable” SUS score for a product to have, or rather whether a product requires more attention and continued improvement, the score is further classified by acceptability ranges—that is, not acceptable (0-50), low/high marginal (51-69), and acceptable (70 and above; Figure 2); higher product acceptability means fewer usability difficulties experienced by a user [19].

Figure 1. Twazon app interface (from top left, clockwise): Twazon dashboard, educational tool; food label tips, and healthy food palm.
View this figure
Figure 2. Grade rankings of System Usability Scale (SUS) scores. Adapted from Bangor et al [18,19].
View this figure

The SUS questions were ranked according to a 5-point Likert scale [20]. Each of the 10 questions had a score range set from 0 to 4. For responses 1, 3, 5, 7, and 9, the score was calculated by subtracting 1 from the scale value. For responses 2, 4, 6, 8, and 10, the score was calculated by subtracting the scale value from 5. The overall score was the total of the scores multiplied by 2.5 [18]. To identify which features of the app participants believed to be the most efficient and which could be improved upon, the participants were asked 2 open-ended questions that were added to the SUS questionnaire but analyzed separately: “What part of the app do you feel works the best?" and “Are there any parts of the app that you feel could be improved, and how?” The results obtained from the 2 additional questions were used to generate visual representations (word clouds) with free online software (Wordle [21]. Word clouds has predominantly been found in social and commercial settings; however, studies have shown that their use in analysis provides “a rapid and practical way to analyse textual data” and helps in “reducing the textual data without bias” [22].

This process allows the reader to quickly identify the most commonly used terms or responses in a given text as it entails illustrating a set of related tags or words in which frequency of word use is reflected visually through font size [23]; this represents the number of participants who gave a response, rather than the total number of responses, and is vital to eliminating the possibility of the same or similar comments being counted multiple times during individual interviews. The answers were sorted into 2 groups (app preferences and app improvements) and a word cloud was generated showing common themes for each question, for a total of 2 word clouds.

The collective results were prepared for analysis in a cross-comparative table using the 13 evidence-informed practice requirements. The information collected in the screening phase was used as a basis for identifying whether these linked to user expectations or experiences.

The Twazon weight loss app intervention was completed by 40 Saudi women with overweight or obesity over the course of 4 months; the rate of attrition was 83%. For the analysis that follows, only the data for the engaged participants (n=26) were used.

System Usability Score (Twazon Intervention)

The overall mean SUS score was 69.3 (SD 10.1), equating to an adjective rating of ok (average=50.9), which suggests that the participants found the app to be more than satisfactory. When compared with the averages for each adjective rating, however, this study’s scores were closer to a rating of good (average=71.4); this translates to an overall acceptability that ranges between high marginal and acceptable (see Figure 2). The highest-rated positive statements responses were numbers 7 (“I imagine most people would learn to use this app very quickly.”) and 9 (“I felt very confident using the app.”). The lowest-rated negative statements were numbers 10 (“I needed to learn a lot of things before I got going with this app.”) and 6 (“I thought there was too much inconsistency in this app.”); see Figure 3.

Word Clouds (Twazon Intervention)

We generated 2 word clouds for the responses given for the 2 open-ended questions regarding the features that were most preferred (question 1) and those that were in need of improvement (question 2). The results for question 1 (Figure 4) showed that the most favored features of the app were the calorie counter, followed by the physical activity calorie calculator and the step counter. The water counter was the fourth most favored feature. The results for question 2 (Figure 5) showed that the primary suggested improvements were to have more food items, followed by change nothing, and then to add more reminders, arrange food items into groups, and social network development.

Figure 3. Mean System Usability Scale (SUS) scores corresponding to the 10 questions. Odd-numbered questions indicate a positive response, while even-numbered questions indicate a negative response. Higher numbers indicate increasing degrees of participant agreement.
View this figure
Figure 4. Word cloud representation of responses indicating the most preferred features.
View this figure
Figure 5. Word cloud representation of responses indicating features in need of improvement.
View this figure


The results of the screening indicated that Arabic weight loss apps had a very low adherence to evidence-informed practices (median=1); no apps had more than 6 evidence-informed practices, and only 9 apps had 4 to 6 integrated, which justified the need to develop an evidence-informed Arabic app. The focus group discussions then led to exploring potential users’ expectations of an ideal app. The results from those discussions indicated that the participants expected all 13 evidence-informed practices to be present in some feature of an ideal app, in addition to its being culturally adapted in terms of language, food, and exercise options (Table 1).

The results obtained from the SUS questionnaires and word clouds indicated that the Twazon users preferred and found useful more than half (7/13) of the evidence-informed practices that were integrated into the apps’ features. The participants reported that 2 of the 13 practices were insufficient (weight loss goal of 0.5-1 kg/week and social network), while they did not mention the remaining practices. A cross-comparative analysis (Table 1) highlighted the relationship between adherence to evidence-informed practices and overall usability of the Twazon weight loss app.

Table 1. Cross-comparative analysis of triangulation exploring potential users’ expectations of an ideal app. N/A: not applicable. NC: no comment.
Evidence-informed practicesArabic weight loss apps (adherence; n=65), n (%)Focus group discussion (expectations)Twazon intervention (experiences)
1. Meal planning25 (38)YesYes
2. Assessing your weight17 (26)YesNC
3. Regular physical activity13 (20)YesYes
4. Maintaining calorie balance10 (15)YesYes
5. Keeping a food diary9 (14)YesYes
6. Portion control0 (0)YesNC
7. Eating a diet rich in fruits and vegetables7 (11)YesYes
8. Tracking your weight6 (9)YesNC
9. Keeping a physical activity journal6 (9)YesYes
10. Weight loss goal of 0.5-1 kg/week5 (8)YesNo
11. Social support2 (3)YesNo
12. Reading nutrition facts labels2 (3)YesNC
13. Water instead of soda/juice7 (11)YesYes
Additional: culturally sensitiveN/AYesNC
Additional: notificationsN/AN/ANo

Principal Findings

The Twazon app aimed to fill a gap in the research and development of evidence-informed Arabic weight loss apps [12] and interventions in order to find the optimum balance between evidence requirements and user needs. The participants’ experiences with the Twazon intervention provided insight into the features of the app that were the least interesting, the most effective, or in need of improvement. When considering the results from the triangulation analysis, a relationship emerged between what is perceived as the best or required practice from the evidence and what participants actually experienced and reported as being useful. Although the Arabic apps failed in general to meet requirements for all 13 evidence-informed practices, the women who took part in the focus group discussions [16] clearly communicated their expectation that all of them should be integrated into an ideal weight loss app.

Some of the evidence-informed practices, such as assessing one’s weight and tracking one’s weight, were not featured in the word clouds as being favored or in need improvement; this could be attributed to their essential nature in general weight loss programs and apps. The practice of eating a diet rich in fruits and vegetables and the practice of reading nutrition facts labels were also recommended by the app; neither was reported as favored or in need of improvement. Analysis of the SUS score for the question regarding the consumption of fruits and vegetables (AA et al, unpublished data, 2017), however, showed that the participants were successful in increasing their intake. This suggests that the app was effective in promoting this diet modification and practice.

Portion control was also recommended by the app, but the results from the word clouds gave no indication that this was either favored or in need of improvement. This could be attributed to a lack of typical serving sizes, which are found in other countries or in other databases. The development of the Twazon app included the creation of the Twazon Saudi Food Database (AA et al, unpublished data, 2017) with the goal of providing users with a detailed list of household measurements for local and international foods to help promote portion size awareness. However, the portion control feature was not mentioned by participants, implying a need for further investigation into the most effective manner in which it should be implemented.

The most commonly user-reported preferences and proposed improvements suggested that the users were more satisfied with the functions of the app (eg, counters) than with the content (eg, missing food item information).

Primary Preferences

The results showed that the most favored features of the app were related to counters (Figure 4). The 4 evidence-informed practices that fulfill the reported preferences for most favored features are maintaining calorie balance (calorie counter), engaging in regular physical activity (step counter), keeping a physical activity and food journal (physical activity calorie calculator and calorie calculator), and drinking water instead of soda or juice (water counter). Our findings contrast with a recent qualitative study of 24 volunteers that suggested that counters are generally not preferred [24].

These results could be due to the fact that weight loss apps, and more specifically Arabic language apps that are culturally adapted, are relatively new to the region [25] and may be considered a novelty. A quote from one of the participants using the Twazon app illustrates this: “I was using an English weight loss app and against my better judgement I opted to eat pizza and burgers instead of kapsa or jarish so that I could count my calories with this app.” The act of counting calories may have been preferred in this study as a result of the participants’ interest in being able to log foods that they were familiar with due to their accessibility in the Twazon Saudi Food Database (AA et al, unpublished data, 2017). In future app development for the Gulf region, counters such as those found in the Twazon app could potentially be useful, as long as the practice remains novel.

Primary Improvements

The primary suggested improvements were to have more food items, followed by have more reminders, arrange food items into groups, and social network development (Figure 5). The second most reported improvement was to change nothing; possible reasons for this were that the users were satisfied with the app, they found the app to be better than their SUS score suggested, or that they simply didn’t report accurately. The suggestions for improvements offer an opportunity to reevaluate the features and structure of the app, with the aim being to inform future app development.

Improvements were mentioned in regard to having access to more food items that are ideally arranged into specific groups to allow users to better log their daily consumption. The users’ adoption of evidence-informed practices, such as meal planning and portion control, may have been hindered by not being able to enter or find certain foods with ease. However, the participants’ inability to report their energy intake could be attributed to a falsely perceived lack of information. In some instances, participants were entering misspelled food items, causing duplicates, or were entering lengthy descriptions of dishes instead of simple keywords; this complicated the task and made logging foods more demanding.

Food data input challenges could be overcome with the addition of a barcode feature, which was one of the three least-reported suggested improvements (see Figure 5). One qualitative study [26] showed that a barcode feature should be considered, as it might improve users’ overall opinion of the quantity and types of food items available, enabling users to update their food intake with the ease of scanning food labels. Future apps might then consider expanding the Twazon Saudi Food Database to include more foods, integrate an autocorrect feature for spelling issues, and offer a barcode scanner to simplify food data input. Further investigation into these features and the user’s perception of them is needed to test their efficacy prior to carrying out an intervention.

The Twazon app provided users with three different types of notifications: (1) tailored tips based on unmet goals in food groups and physical activity, (2) general tips for foods to consume and foods to avoid, and (3) a reminder to enter weight and fill in the food palm tree assessment [8]; we gave them the option to choose how often (every 2 days, every 3 days, and every week) they received the first and second types of notifications, but the third type was automatically delivered every 2 weeks on completion of the required input. Although this was done to avoid overwhelming the participants, one study by Freyne et al [27] found that 3 notifications daily did not frustrate the users, exemplifying that an increase in notifications is not necessarily a hindrance. We suggest that more communicative contact should be considered in the development of future weight loss apps so as to encourage users to record as much as possible.

The evidence-informed practice of having a minimum weight loss goal of 0.5 to 1 kg/week perhaps identified the greatest relationship; many women reported losing interest in participating in the intervention due to not being satisfied with the aforementioned goal. This outcome could be explained by an aversion to goals perceived as being impossible or unsatisfying [28]. Several studies showed that participants with obesity are not motivated by an overall weight loss goal of 5% to 10%, as is recommended by health professionals, but rather a weight loss goal of 22% to 34% [29-31]. This failure to meet the expectations of patients with obesity suggests that there needs to be a smaller disparity between actual and expected outcomes. If this is achieved, then the probability of negative effects that are seemingly caused by unmet expectations can be lessened, and in turn more positive weight loss outcomes [32] can be achieved.

Evidence-informed weight loss programs have suggested that social networking could have a positive effect on weight loss outcomes; social media-based reports and sharing via social media sites such as Twitter are effective in weight loss interventions [33], as they can help motivate and empower participants to work harder toward their goals. In the Twazon app, we created an original and private social network that was accessible solely to the users of the app. However, the intervention participants reported the need for more social network development. Despite the remarkably high rate of social media use in the region, it seems that Saudi female participants were not as inclined as expected to share and interact in regard to their weight loss experience within the closed group.

Lack of engagement with the social media aspect of the app could be attributed to the participants not having direct support from family and friends, as is typical on most popular social media sites; the Twazon app was accessible exclusively to registrants of the app. However, one study [34] found no significant differences in a 6-month weight loss intervention between 3 different groups, which included a podcast plus Twitter group. Regardless, our results from the word clouds show that participants desired more social media development, suggesting that the use of social media sites as a tool to help promote weight loss and connect weight loss intervention participants should be considered and optimized in future app-based interventions.

Despite the integration of evidence-informed features into the Twazon app, challenges with retention still arose. To improve retention in future app interventions of this kind, modifications to the social networking feature and an increase in the amount of contact with the user is highly recommended. The reported user experiences also suggest that more consideration needs to be given to establishing weight loss goals that are not demotivating in order to facilitate more successful weight loss outcomes.


Participants deemed the Twazon app to be of acceptable usability. The triangulation analysis revealed the greatest relationship to be the disparity between user experience and 2 of the evidence-informed practices, namely, a minimum weight loss goal of 0.5 to 1 kg/week and social support. In contrast, user expectations coincided with evidence-informed practices and therefore did not provide any relationship. Once the aforementioned improvements are made, it would be feasible for health care providers to recommend the use of Twazon in weight loss programs that involve behavioral modification strategies. Further in-depth exploration through qualitative study is also needed to better understand the relationship observed so as to appeal to the motivating factors that drive participants toward successful outcomes in their weight loss goals when using weight loss apps.


This research project was supported by a grant from the Research Center of the Female Scientific and Medical Colleges, Deanship of Scientific Research, King Saud University, Riyadh, Saudi Arabia.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Tools in Twazon addressing evidence-informed weight loss practices.

PDF File (Adobe PDF File), 34KB

  1. ALNohair S. Obesity in gulf countries. Int J Health Sci (Qassim) 2014 Jan;8(1):79-83 [FREE Full text] [Medline]
  2. Bert F, Giacometti M, Gualano MR, Siliquini R. Smartphones and health promotion: a review of the evidence. J Med Syst 2014 Jan;38(1):9995. [CrossRef] [Medline]
  3. Pagoto S, Schneider K, Jojic M, DeBiasse M, Mann D. Evidence-based strategies in weight-loss mobile apps. Am J Prev Med 2013 Nov;45(5):576-582. [CrossRef] [Medline]
  4. Chen J, Cade JE, Allman-Farinelli M. The most popular smartphone apps for weight loss: a quality assessment. JMIR Mhealth Uhealth 2015 Dec 16;3(4):e104 [FREE Full text] [CrossRef] [Medline]
  5. Nikolaou CK, Lean ME. Mobile applications for obesity and weight management: current market characteristics. Int J Obes (Lond) 2017 Dec 13;41(1):200-202 [FREE Full text] [CrossRef] [Medline]
  6. Schnall R, Rojas M, Bakken S, Brown W, Carballo-Dieguez A, Carry M, et al. A user-centered model for designing consumer mobile health (mHealth) applications (apps). J Biomed Inform 2016 Apr;60:243-251 [FREE Full text] [CrossRef] [Medline]
  7. KSA ICT indicators end of Q3 2016. Riyadh, Kingdom of Saudi Arabia: Communications and Information Technology Commission; 2016.   URL: [accessed 2017-12-28] [WebCite Cache]
  8. Alnasser A, Sathiaseelan A, Al-Khalifa A, Marais D. Development of 'Twazon': an Arabic app for weight loss. JMIR Res Protoc 2016 May 16;5(2):e76 [FREE Full text] [CrossRef] [Medline]
  9. Azar KMJ, Lesser LI, Laing BY, Stephens J, Aurora MS, Burke LE, et al. Mobile applications for weight management: theory-based content analysis. Am J Prev Med 2013 Nov;45(5):583-589. [CrossRef] [Medline]
  10. Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res 2017 Dec 26;17(1):88 [FREE Full text] [CrossRef] [Medline]
  11. US Department of Health and Human Services. What & why of usability: glossary. Washington, DC: US Department of Health & Human Services; 2018.   URL: [accessed 2018-03-03] [WebCite Cache]
  12. Alnasser AA, Amalraj RE, Sathiaseelan A, Al-Khalifa AS, Marais D. Do Arabic weight-loss apps adhere to evidence-informed practices? Transl Behav Med 2016 Dec;6(3):396-402 [FREE Full text] [CrossRef] [Medline]
  13. Centre for Public Health Excellence at NICE. Obesity: the prevention, identification, assessment and management of overweight and obesity in adults and children. National Institute for Health and Clinical Excellence 2006 Dec. [Medline]
  14. Logue J, Thompson L, Romanes F, Wilson DC, Thompson J, Sattar N, Guideline Development Group. Management of obesity: summary of SIGN guideline. BMJ 2010 Feb 24;340:c154. [Medline]
  15. Australian Government, National Health and Medical Research Council, Department of Health and Ageing. Clinical practice guidelines for the management of overweight and obesity in adults, adolescents and children in Australia. Canberra, Australia: Commonwealth of Australia; 2013 May.   URL: [accessed 2017-12-28] [WebCite Cache]
  16. Alnasser AA, Alkhalifa AS, Sathiaseelan A, Marais D. What overweight women want from a weight loss app: a qualitative study on arabic women. JMIR Mhealth Uhealth 2015 May 20;3(2):e41 [FREE Full text] [CrossRef] [Medline]
  17. Nielsen J. Why you only need to test with 5 users. Fremont, CA: Nielsen Norman Group; 2000 Mar 19.   URL: [accessed 2017-12-28] [WebCite Cache]
  18. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009;4(3):114-123 [FREE Full text]
  19. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the System Usability Scale. Int J Hum Comput Interact 2008;24(6):574-594.
  20. Brooke J. SUS-a quick and dirty usability scale. In: Patrick W, Jordan B, Thomas IL, editors. Usability Evaluation In Industry. London, UK: Taylor & Francis; 1996:189-194.
  21. Feinberg J. Wordle. 2014.   URL: [accessed 2018-03-27] [WebCite Cache]
  22. Gill D, Griffin A. Good medical practice: what are we trying to say? Textual analysis using tag clouds. Med Educ 2010 Mar;44(3):316-322. [CrossRef] [Medline]
  23. Prochaska JJ, Fromont SC, Wa C, Matlow R, Ramo DE, Hall SM. Tobacco use and its treatment among young people in mental health settings: a qualitative analysis. Nicotine Tob Res 2013 Aug;15(8):1427-1435 [FREE Full text] [CrossRef] [Medline]
  24. Solbrig L, Jones R, Kavanagh D, May J, Parkin T, Andrade J. People trying to lose weight dislike calorie counting apps and want motivational support to help them achieve their goals. Internet Interv 2017 Mar;7:23-31 [FREE Full text] [CrossRef] [Medline]
  25. Mansar SL, Jariwala S, Shahzad M, Anggraini A, Behih N, AlZeyara A. A usability testing experiment for a localized weight loss mobile application. Procedia Technol 2012;5:839-848. [CrossRef]
  26. Tang J, Abraham C, Stamp E, Greaves C. How can weight-loss app designers best engage and support users? A qualitative investigation. Br J Health Psychol 2015 Feb;20(1):151-171. [CrossRef] [Medline]
  27. Freyne J, Yin J, Brindal E, Hendrie GA, Berkovsky S, Noakes M. Push notifications in diet apps: influencing engagement times and tasks. Int J Hum Comput Interact 2017 Jan 11;12:1-3. [CrossRef]
  28. Foster GD, Wadden TA, Phelan S, Sarwer DB, Sanderson RS. Obese patients' perceptions of treatment outcomes and the factors that influence them. Arch Intern Med 2001 Sep 24;161(17):2133-2139. [Medline]
  29. Foster GD, Wadden TA, Vogt RA, Brewer G. What is a reasonable weight loss? Patients' expectations and evaluations of obesity treatment outcomes. J Consult Clin Psychol 1997 Feb;65(1):79-85. [Medline]
  30. Jeffery RW, Wing RR, Mayer RR. Are smaller weight losses or more achievable weight loss goals better in the long term for obese patients? J Consult Clin Psychol 1998 Aug;66(4):641-645. [Medline]
  31. O'Neil PM, Smith CF, Foster GD, Anderson DA. The perceived relative worth of reaching and maintaining goal weight. Int J Obes Relat Metab Disord 2000 Aug;24(8):1069-1076. [Medline]
  32. Leung KK, Silvius JL, Pimlott N, Dalziel W, Drummond N. Why health expectations and hopes are different: the development of a conceptual model. Health Expect 2009 Dec;12(4):347-360 [FREE Full text] [CrossRef] [Medline]
  33. Dahl AA, Hales SB, Turner-McGrievy GM. Integrating social media into weight loss interventions. Curr Opin Psychol 2016 Jun;9:11-15. [CrossRef]
  34. Kozak AT, Buscemi J, Hawkins MAW, Wang ML, Breland JY, Ross KM, et al. Technology-based interventions for weight management: current randomized controlled trial evidence and future directions. J Behav Med 2017 Feb;40(1):99-111. [CrossRef] [Medline]

SUS: System Usability Scale

Edited by G Eysenbach; submitted 04.01.18; peer-reviewed by S Barbieri, JY Kim; comments to author 23.01.18; revised version received 03.03.18; accepted 19.03.18; published 17.04.18


©Aroub Alnasser, Janet Kyle, Abdulrahman Alkhalifah, Debbi Marais. Originally published in JMIR Human Factors (, 17.04.2018.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.