Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, December 24 through Wednesday, December 26 inclusive. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 21.12.18 in Vol 5, No 4 (2018): Oct-Dec

This paper is in the following e-collection/theme issue:

    Original Paper

    The Service User Technology Acceptability Questionnaire: Psychometric Evaluation of the Norwegian Version

    1Department of Nursing and Health Promotion, Faculty of Health Sciences, OsloMet – Oslo Metropolitan University, Oslo, Norway

    2General Practice Research Unit, Department of General Practice, Institute of Health and Society, University of Oslo, Oslo, Norway

    3Norwegian Centre for E-health Research, University Hospital of North Norway, Tromsø, Norway

    4Department of Clinical Medicine, Faculty of Health Sciences, UiT The Arctic University of Norway, Tromsø, Norway

    Corresponding Author:

    Astrid Torbjørnsen, MSc

    Department of Nursing and Health Promotion

    Faculty of Health Sciences

    OsloMet – Oslo Metropolitan University

    Postboks 4 St. Olavs plass

    Oslo, 0130


    Phone: 47 92633075



    Background: When developing a mobile health app, users’ perception of the technology should preferably be evaluated. However, few standardized and validated questionnaires measuring acceptability are available.

    Objective: The aim of this study was to assess the validity of the Norwegian version of the Service User Technology Acceptability Questionnaire (SUTAQ).

    Methods: Persons with type 2 diabetes randomized to the intervention groups of the RENEWING HEALTH study used a diabetes diary app. At the one-year follow-up, participants in the intervention groups (n=75) completed the self-reported instrument SUTAQ to measure the acceptability of the equipment. We conducted confirmatory factor analysis for evaluating the fit of the original five-factor structure of the SUTAQ.

    Results: We confirmed only 2 of the original 5 factors of the SUTAQ, perceived benefit and care personnel concerns.

    Conclusions: The original five-factor structure of the SUTAQ was not confirmed in the Norwegian study, indicating that more research is needed to tailor the questionnaire to better reflect the Norwegian setting. However, a small sample size prevented us from drawing firm conclusions about the translated questionnaire.

    JMIR Hum Factors 2018;5(4):e10255




    Patients’ perceptions are important components of any health technology assessment when developing and introducing technological devices for self-management. Scientific and robust methods are necessary in the evaluation of the technology, including the use of a framework such as the Model of Assessment of Telemedicine [1,2].

    In previous research, both qualitative and quantitative research methods and log data from self-monitoring have been used in the evaluation of acceptability. Many published studies use questionnaires [3,4], which are often self-constructed and not validated [4], making the comparison of results across studies difficult. Further, many of these studies are small, with few participants, and have methodological limitations [4]. In particular, limitations related to the development phase and psychometric evaluation of questionnaires measuring patient satisfaction are present, with evaluations lacking data on factor structures, reliability, and validity [5].

    There is no consensus related to the definition of the acceptability in mobile health (mHealth) research, although a long list of definitions exists, combining technology and health [6] with users’ perspectives [7]. Previous research has defined users’ perspectives within telemedicine as “issues related to the perception of the patient or the relatives of the telemedicine application including the patients’ and relatives’ acceptance of the technology” [1]. However, we have not been able to find the user perspective defined in terms of mHealth. The acceptability of digital solutions in health care is often used synonymously with the concept of satisfaction [7]. In the development of the acceptability questionnaire Service User Technology Acceptability Questionnaire (SUTAQ), Hirani et al aimed to investigate the concept of technology acceptance in more detail [8].

    The aim of this study was to assess the validity of the translated Norwegian version of the SUTAQ acceptability questionnaire. This was tested on participants who used an mHealth tool, namely, a digital diabetes diary app running on a mobile phone and a blood glucose meter transferring blood glucose measurements to the app by Bluetooth in the intervention groups of a randomized controlled trial (RCT).


    European Union Project

    The European Union (EU) project, REgioNs of Europe WorkINg toGether for HEALTH (RENEWING HEALTH), was a research collaboration between 9 regions in Europe working with designing and implementing telemedicine services. The data used in this paper were drawn from the Norwegian study that was a part of this EU project. The acceptability of the equipment was measured at the one-year follow-up in an RCT (NCT01315756).

    Participants and Setting

    Persons with type 2 diabetes were randomized to 3 groups. The 2 intervention groups received a diabetes diary app that they had for 1 year, and one of the groups also received health counseling for the first 4 months. In addition, the study had a control group. The participants lived at home and were recruited from primary health care. Of the 101 participants who were randomized to the 2 intervention groups, 74.3% (75/101) completed the SUTAQ questionnaire. Other results from the RCT are reported in detail elsewhere [9-12].

    Service User Technology Acceptability Questionnaire

    The SUTAQ was developed for the Whole Systems Demonstrator (WSD) study in the United Kingdom, to measure acceptability and identify the characteristics of persons who were likely to reject technological health services (see Multimedia Appendix 1) [8]. The questionnaire has 22 items, measured on a Likert-scale from 1 to 6, reflecting more or less agreement with the item statements, respectively. The questionnaire has 5 subscales, where each contains between 3 and 9 items. The subscale containing 9 items was further divided into 2. The original items and the subscales are presented later in the paper. The original questionnaire was found to be reliable and valid [8].

    As the partners in the RENEWING HEALTH study in 2011 had decided to include answers to SUTAQ in the minimum common dataset, the questionnaire was also used in the Norwegian trial, even though our data collection had already started. The questionnaire was not available in Norwegian when this study started. However, the translation process followed the procedure recommended by the European Organization for Research and Treatment of Cancer Quality of Life Group [13] and the published guidelines for cognitive interviews [14,15]. Two professional translators translated the SUTAQ questionnaire from English to Norwegian. The Norwegian research team considered the discrepancy between the 2 translated versions and the English version. We achieved equivalence with regard to aspects such as the meaning of words, expressions, concepts, and cultural context. A cultural adaptation of the questionnaire had to be done only for a few statements.

    A native English speaker, a bilingual person, without any initial knowledge of the SUTAQ, backward translated the final Norwegian version. The research team, also with a good command of English, compared the backward translation with the original questionnaire, and no further changes were made.

    Finally, we conducted cognitive interviews with 10 random participants who had answered the SUTAQ questionnaire. According to these interviews, the items were understandable to the participants, although some found the language somewhat cumbersome, leading us to make a few adjustments.

    The report from the translation process can be obtained from the last author (LR).

    Statistical Analysis

    The sample was described using descriptive statistics. To assess the construct validity of the present domains in the SUTAQ questionnaire from the WSD study, we conducted a confirmatory principal component factor analysis on the 22 items, with Varimax rotation and with a fixed number of 5 factors in accordance with the WSD study [8]. To assess the internal consistency of each domain or extracted factor and for the entire questionnaire, we calculated Cronbach alphas. All analyses were performed using IBM SPSS Statistics v23 (IBM Corp, Armonk, NY, USA).


    Sample Characteristics

    In total, we analyzed data from 75 participants, of whom 56% (42/75) were female. The age range was 35-80 years, with a median age of 59 years, and 49% (37/75) had ≥12 years of education. There were no differences between the 2 intervention groups for the SUTAQ findings. We found no differences in the baseline measures between the 75 participants included in the analyses and the 26 who dropped out during the study. More details concerning demographic and clinical results from the study sample are published elsewhere [16].

    The median values for the original SUTAQ domains are presented in Figure 1, indicating that the participants accepted the equipment to a high degree within the 3 areas of privacy and discomfort, care personnel concerns, and satisfaction. This implies a high degree of acceptability regarding beliefs about the security of the monitored data, the impact of the equipment on the user, beliefs of the continuity and skills of the health care personnel facilitating the equipment, and acceptance and satisfaction with the equipment and the given service. The median value between 1 and 6 constitutes the middle value in the figure. The two categories, privacy and discomfort and care personnel concerns are based on items with negative statements, where high values reflect a high degree of agreement with the negative statements in these two categories, which means that low values represent a positive score. The remaining factors consist of positive statements. High values reflect a high degree of agreement. The participants reported being slightly more than medium positive concerning whether the equipment could improve their care or increase their access to health care within the domain perceived benefit. Results from the domain kit as substitution indicated that the participants were most critical about the statements concerning this digital solution replacing usual care.

    Factorial Reliability and Validity

    The measurement properties of the SUTAQ are presented in Table 1. Overall, the amount of missing data was minimal, no more than 8% for all items. The floor effect was small; only 4 items were far above 15%, considered to be problematic [17]. However, the number of items with ceiling effects was higher, with only about half of the items below the limit of 15%, and for 5 of the items, around 50% (34-40/75) of the participants reached the highest possible score.

    The confirmatory factor analysis revealed that only factor 1 and factor 3 were consistent in the original study and this study (Table 2). The first factor, Perceived benefit, had 9 items in the original factor structure. Of the items in the Norwegian dataset, 7 loaded >0.400, which was the limit within the factors in the WSD study [8]. In the third domain, Care personnel concerns, all 3 items loaded >0.400. The Cronbach alpha coefficient for all 22 items was .851, which demonstrates good internal consistency [18]. Cronbach alpha values for each factor are listed in Table 2.

    Figure 1. Median reported scores of the Service User Technology Acceptability Questionnaire domains.
    View this figure
    Table 1. Service User Technology Acceptability Questionnaire item descriptors.
    View this table
    Table 2. Confirmatory factor analysis showing Cronbach alpha values.
    View this table


    Principal Findings

    The Norwegian version of SUTAQ revealed good internal consistency, with a Cronbach alpha of .851. However, the original five-factor solution was not confirmed. On the contrary, our results indicated that a one-factor solution, or at most a three-factor solution, was sufficient, as the explained variance increased by <6% when adding more factors (Table 2). Moreover, only 2 items were loaded on each of the last factors (factors 4 and 5), indicating that they were superfluous. In addition, we found that the SUTAQ questionnaire had some items with a floor effect and even more items with ceiling effects.


    One limitation of this study was the low number of participants, as over 250 or at least 10 participants per item is recommended to enable precise conclusions from factor analysis [19]. Further, a factor loading above 0.7 per item is preferred according to Kaiser’s criteria [20]. Thus, the small sample size might be one of the possible explanations for the lack of confirmation of all factors. Exploratory factor analysis would have been a suitable statistical method to explore the potential of the questionnaire in our Norwegian setting, although demanding a larger number of participants.

    Differences in study contexts, health issues, and equipment could also contribute to the lack of common factors in the original study and this study. In the WSD study, interventions were given to patients with long-term conditions, not only diabetes but also chronic obstructive pulmonary disease, heart failure, and social needs [21]. Further, a far broader range of equipment was used in the WSD study: both telehealth and telecare. In this study, only persons with type 2 diabetes used the self-management app, and no telemonitoring was involved. Outdated equipment was also a problem in the Norwegian study because of a long inclusion process [10].

    Our data were slightly skewed (Table 1), and to our knowledge, there are no references to an acceptable level of floor and ceiling effects in similar technological studies. Quality criteria available in the literature suggest that floor or ceiling effects over 15% will reduce the reliability of the item in health status questionnaires. In addition, such an item cannot distinguish between the groups of responders scoring at either end of the scale [17]. Only 6 of the 22 items had an acceptable level (≤15%) of both floor and ceiling effects. Other SUTAQ studies [8,22] did not report on the floor and ceiling effects of each item but did present histograms and means for the domains. It seems that the data on the domains Satisfaction and Privacy and discomfort were skewed in those studies [8,22]. Hirani et al [8] explained the skewedness of items as being linked to the dropout rate from their study, as persons dropping out could have scored somewhat different from the remaining participants, possibly leading to bias and reduced generalizability. The responders were expected to be more satisfied than nonresponders; this explanation could also be relevant for our Norwegian study. However, even if the remaining participants were more satisfied, the questionnaire did not capture details of their satisfaction.

    Using an unvalidated questionnaire is a limitation as described by Streiner [18]. This refers both to the development of the questionnaire and to the generalizability of the translated version, which may lack equivalence with the original questionnaire. Being part of a large EU study, we agreed upon the selection of common questionnaires. Before our one-year follow-up, the partners decided to introduce the SUTAQ. At that time, we translated the instrument according to standardized procedures for translation [13]. This gave us knowledge about the participants’ conceptual and semantic understanding of the items. If we had the opportunity to perform a questionnaire validation of the SUTAQ ahead of the study, this would have improved reflections about its validity. Another aspect is that SUTAQ was developed for the WSD study evaluating different technologies and measuring the acceptability of telehealth and telecare interventions, with a closer follow-up from health care personnel than that in the Norwegian self-management study. The differences in the content of the interventions between the original [8] and this mHealth study could have affected the validation analysis, as the SUTAQ might be more suitable for a different type of intervention than the one implemented in this study. Finally, even though we carefully followed the translation procedures, we cannot rule out the risk that the translation from English to Norwegian could have changed the understanding of the initial meaning of the statements in SUTAQ.

    Originally, we aimed to perform a test-retest analysis to measure reliability, which would require data on 40-50 participants. Unfortunately, we did not reach the sufficient number of participants because of financial and logistical difficulties. We measured acceptability at the last point of follow-up in the study, making it difficult to collect additional retest questionnaires. Given that we had only 12 retest responders, we realized that we did not have enough statistical power to perform a meaningful test-retest analysis.

    Implications for Future Research and Clinical Practice

    In the diverse reality of technology and health, it is challenging to measure patient perception. Nevertheless, we are still in need of a questionnaire that measures the acceptability of digital interventions, given the current development and implementation of many new apps and Web solutions in health care. Health technology assessment as a systematic evaluation contributes to the evaluation of various impacts of health technology [23], so there is a need for validated measurements of the acceptability of the technology among users. The SUTAQ measures several such relevant aspects, such as the impact on relations to health care personnel, privacy, etc. A relatively small sample size has restrained us from drawing any firm conclusions. SUTAQ should be validated using a larger sample and possibly a modified version developed for use in the Norwegian setting.


    This Norwegian project was funded by the EU and the Information and Communication Technologies Policy Support Programme and Innovation Framework Program, the Norwegian Research Council, the Health Authorities of Northern Norway, the Norwegian Centre for E-health Research, Oslo and Akershus University College of Applied Sciences, Akershus University Hospital, and the Norwegian Diabetes Association.

    We thank all participants, general practitioners, nurses, and office staff who helped us perform the study. We also thank our colleagues, Astrid Grøtland (project leader), Tone Singstad (diabetes specialist nurse), Elisabeth Elind (nutritionist), and Heidi Holmen. Furthermore, we would like to thank Elisabeth Ellefsen Sjåeng, Erlend Bønes, and Kamran Bagheri for technical support.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    The Service User Technology Acceptability Questionnaire (original version; published with permission from Shashi Hirani).

    PDF File (Adobe PDF File), 402KB


    1. Kidholm K, Ekeland AG, Jensen LK, Rasmussen J, Pedersen CD, Bowes A, et al. A model for assessment of telemedicine applications: mast. Int J Technol Assess Health Care 2012 Jan;28(1):44-51. [CrossRef] [Medline]
    2. Kidholm K, Jensen L, Kjølhede T, Nielsen E, Horup M. Validity of the Model for Assessment of Telemedicine: A Delphi study. J Telemed Telecare 2018 Feb;24(2):118-125. [CrossRef] [Medline]
    3. Harrison S, Stadler M, Ismail K, Amiel S, Herrmann-Werner A. Are patients with diabetes mellitus satisfied with technologies used to assist with diabetes management and coping?: A structured review. Diabetes Technol Ther 2014 Nov;16(11):771-783. [CrossRef] [Medline]
    4. Holmen H, Wahl A, Cvancarova Småstuen M, Ribu L. Tailored Communication Within Mobile Apps for Diabetes Self-Management: A Systematic Review. J Med Internet Res 2017 Jun 23;19(6):e227 [FREE Full text] [CrossRef] [Medline]
    5. Allemann IM, Kunz R, Blozik E. Instruments to assess patient satisfaction after teleconsultation and triage: a systematic review. Patient preference and adherence 2014;8:a. [Medline]
    6. Oh H, Rizo C, Enkin M, Jadad A. What is eHealth (3): a systematic review of published definitions. J Med Internet Res 2005 Feb 24;7(1):e1 [FREE Full text] [CrossRef] [Medline]
    7. Kidholm K, Nielsen A, Prior R. 2 Draft Questionnaire for data collection. Document D3   URL: http:/​/www.​​documents/​28946/​394215/​D3.​2+v1.​0+Renewing+Health+Draft+Questionnaire+for+data+collection.​pdf [accessed 2017-06-21] [WebCite Cache]
    8. Hirani SP, Rixon L, Beynon M, Cartwright M, Cleanthous S, Selva A, et al. Quantifying beliefs regarding telehealth: Development of the Whole Systems Demonstrator Service User Technology Acceptability Questionnaire. J Telemed Telecare 2017 May;23(4):460-469. [CrossRef] [Medline]
    9. Ribu L, Holmen H, Torbjørnsen A, Wahl AK, Grøttland A, Småstuen MC, et al. Low-intensity self-management intervention for persons with type 2 diabetes using a mobile phone-based diabetes diary, with and without health counseling and motivational interviewing: protocol for a randomized controlled trial. JMIR Res Protoc 2013 Aug 26;2(2):e34 [FREE Full text] [CrossRef] [Medline]
    10. Torbjørnsen A, Jenum AK, Småstuen MC, Arsand E, Holmen H, Wahl AK, et al. A Low-Intensity Mobile Health Intervention With and Without Health Counseling for Persons With Type 2 Diabetes, Part 1: Baseline and Short-Term Results From a Randomized Controlled Trial in the Norwegian Part of RENEWING HEALTH. JMIR Mhealth Uhealth 2014 Dec 11;2(4):e52 [FREE Full text] [CrossRef] [Medline]
    11. Holmen H, Torbjørnsen A, Wahl AK, Jenum AK, Småstuen MC, Arsand E, et al. A Mobile Health Intervention for Self-Management and Lifestyle Change for Persons With Type 2 Diabetes, Part 2: One-Year Results From the Norwegian Randomized Controlled Trial RENEWING HEALTH. JMIR Mhealth Uhealth 2014 Dec 11;2(4):e57 [FREE Full text] [CrossRef] [Medline]
    12. Bradway M, Pfuhl G, Joakimsen R, Ribu L, Grøttland A, Årsand E. Analysing mHealth usage logs in RCTs: Explaining participants' interactions with type 2 diabetes self-management tools. PLoS One 2018;13(8):e0203202 [FREE Full text] [CrossRef] [Medline]
    13. Dewolf L, Koller M, Velikova G, Johnson C, Scott N, Bottomley A. EORTC Quality of Life Group translation procedure. Brussels, Belgium: EORTC Quality of Life Group; 2009.
    14. Nøtnes T. Innføring i kognitiv kartlegging [Introduction to cognitive mapping]. Norway: Statistisk sentralbyrå [Statistics Norway]; 2001 Apr.   URL: [accessed 2017-06-29] [WebCite Cache]
    15. Oltedal S, Garratt A. Videreutvikling av spørreskjema for å måle pasienterfaringer blant inneliggende somatiske pasienter-Dokumentasjonsrapport [Further development of a questionnaire to measure patient experiences among somatic inpatients - A documentation report]. Oslo, Norway: Norwegian Research Centre for Health Services; 2006:1-26.
    16. Torbjørnsen A, Småstuen MC, Jenum AK, Årsand E, Ribu L. Acceptability of an mHealth App Intervention for Persons With Type 2 Diabetes and its Associations With Initial Self-Management: Randomized Controlled Trial. JMIR Mhealth Uhealth 2018 May 21;6(5):e125 [FREE Full text] [CrossRef] [Medline]
    17. Terwee CB, Bot SDM, de Boer MR, van der Windt DAWM, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007 Jan;60(1):34-42. [CrossRef] [Medline]
    18. Streiner DL, Norman GR, Cairney J. Health measurement scales: A practical guide to their development and use. Oxford: Oxford University Press; Jan 2015.
    19. Osborne J, Costello A. Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Pan-Pacific Management Review 2009;12(2):131-146 [FREE Full text]
    20. Pett MA, Lackey NR, Sullivan J. Making sense of factor analysis: the use of factor analysis for instrument development in health care research. Thousand Oaks, California: Sage Publications; 2003.
    21. Bower P, Cartwright M, Hirani SP, Barlow J, Hendy J, Knapp M, et al. A comprehensive evaluation of the impact of telemonitoring in patients with long-term conditions and social care needs: protocol for the whole systems demonstrator cluster randomised trial. BMC Health Serv Res 2011 Aug 05;11:184 [FREE Full text] [CrossRef] [Medline]
    22. Dario C, Luisotto E, Dal Pozzo E, Mancin S, Aletras V, Newman S, et al. Assessment of Patients' Perception of Telemedicine Services Using the Service User Technology Acceptability Questionnaire. Int J Integr Care 2016 Jun 01;16(2):13 [FREE Full text] [CrossRef] [Medline]
    23. World Health Organization. Health technology assessment of medical devices. Geneva: World Health Organization; 2011.


    EU: European Union
    mHealth: mobile health
    RCT: randomized controlled trial
    RENEWING HEALTH: REgioNs of Europe WorkINg toGether for HEALTH
    SUTAQ: Service User Technology Acceptability Questionnaire
    WSD: Whole Systems Demonstrator

    Edited by G Eysenbach; submitted 28.02.18; peer-reviewed by S Li, A Menon; comments to author 17.08.18; revised version received 11.09.18; accepted 14.09.18; published 21.12.18

    ©Astrid Torbjørnsen, Milada C Småstuen, Anne Karen Jenum, Eirik Årsand, Lis Ribu. Originally published in JMIR Human Factors (, 21.12.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.