Self-Determined Health App Evaluation Questionnaire Development: Mixed Methods Study

Introduction

Background

The global as well as national supply of health apps is overwhelming, and more and more people are using health apps. This raises the basic questions of which apps can be considered health apps, and how can the credibility of these apps be better assessed.

Fundamentally, a distinction is made between [1] health and mobile apps. Health apps are applications targeting consumers. They are intended to be used for preventive purposes and help promote or change lifestyle in a health-promoting manner.

On the other hand, medical apps are applications used to support “self-empowerment” (self-determination) and, for example, are supportive in coping with chronic diseases or in rehabilitation [2], which are subject to regulations like the medical device regulations.

The use of evaluation criteria for health apps is important for several reasons, as described below.

Functionality

Evaluation criteria can assess the functionality of health apps to ensure that the intended tasks are effectively fulfilled. This could include monitoring health parameters, reminding users to take medication or providing personalized health recommendations.

Quality assurance

Evaluation criteria help assess the quality of health apps. This is particularly important because the information and functions provided by health apps can have a direct impact on the health of users. Criteria enable ensuring that the apps provide accurate, reliable, and evidence-based information.

Evidence-based content

Health apps should be based on scientific knowledge to provide accurate and reliable information. Evaluation criteria can verify whether the content presented in the apps is based on current scientific evidence.

Usability and user experience

The usability of health apps is crucial for their effectiveness. Evaluation criteria can assess usability and accessibility to ensure that the apps are easily understandable and accessible. Furthermore, accessibility criteria may be important for allowing the apps to be used by impaired users.

Privacy

Apps may process personal health data, so it is crucial to ensure that only the minimal set of data is acquired (“data minimization”) and that data are secured and protected in transit and at rest. Evaluation criteria can consider aspects such as data privacy, encryption, and compliance with regulatory requirements.

Regulatory compliance

Apps, especially health apps may need to comply with certain legal and regulatory requirements. Evaluation criteria can assess whether the apps comply with applicable regulations and standards.

Study Objectives

By applying evaluation criteria, users, health care providers, and regulatory authorities can make more- or better-informed decisions about the selection and use of health apps to ensure that they make a positive contribution to the health and well-being of users. Payors also need clear and transparent criteria to decide on reimbursement aspects.

The objective of this study was to establish evaluation criteria for users to make well-informed decisions in regard to health apps. Health app evaluation frameworks, such as the Mobile App Rating Scale (MARS) [3] and the ENLIGHT Tool [4], provide structured criteria for assessing functionality, usability, and information quality, typically for professional or research use; however, this work focuses on evaluation by laypersons rather than professional organizations, emphasizing user-friendly criteria for informed decision-making.

Methods

Ethical Considerations

Formal ethics approval was not required for this study because no patients or other vulnerable populations were involved. The user survey was completed anonymously by registered participants who volunteered to take part. No identifying information was collected at any stage, and all respondents provided informed consent before proceeding.

Literature Search and Selection of Relevant Documents

The selection of evaluation criteria was conducted as part of a literature search based on the keywords “Bewertungskriterien Apps Gesundheit” (evaluation criteria health apps) and “evaluation criteria health Apps” in English. The search was carried out on the portals Google Scholar and PubMed, and additional literature references from publications were followed up based on the results.

The World Health Organization (WHO) includes “health apps” in the “Classification of Digital Health Interventions v1.0 (DHI)” [5,6]. This primarily categorizes apps according to target groups (eg, for clients or citizens or health care providers) and health system challenges, which then result in system categories for digital health applications.

According to the health system challenges level of health apps, “Lost client data” entails the interventions “1.1.3. Provision of individualized reminder services for clients” and “2.2.1 Longitudinal recording of health status and services of clients”, corresponding on a system level to the functions “Client Information System” and “Electronic Health Record”. The taxonomy used was based on [7] and the “Guidelines for reporting of health interventions using mobile phones: mobile health evidence reporting and assessment (mERA) checklist” [8].

These two documents generally refer to comprehensive mobile health (mHealth) applications with a strong focus on already manifested diseases.

Partially, reliance was placed on systematic reviews already conducted, such as health technology assessment frameworks for eHealth: a systematic review [9] or framework for supporting reimbursement decisions for digital health applications (mHealth) and its (retrospective) application to selected examples [1].

However, all of these encompass digital health applications, not just health apps. The existing assessment frameworks use different formats (checklists, questionnaires, and tables) and show differences in terms of the spectrum of evaluation domains, additional technology-specific aspects, and target groups. None of the frameworks are fully established or field-tested.

At the European level, there is also the initiative Label2Enable [10]. Label2Enable is an EU-funded project. Its task is to introduce CEN-ISO/TS 82304‐2 and its quality seal for health and wellness apps. The goal is to create a digital single market where high-quality health apps can be extensively used in prevention, health care, and self-care.

The ISO/TS 82304-2:2021(en) “Health software — Part 2 [11]: Health and wellness Apps—Quality and reliability” served as the basis for developing the evaluation criteria.

The following publications also served as the basis for the evaluation criteria:

1. Health apps. Basic paper with special consideration of the aspect of health literacy [12],
2. Opportunities and risks of health apps (CHARISMHA); German: chances and risks of mobile health apps (CHARISMHA) [13],
3. Deutsches Netzwerk Versorgungsforschung e. V. Memorandum - health and medical apps [2],
4. Health apps: How can certification look concretely: Learning from experiences here and elsewhere [14],
5. Digitization and patient safety - HE 2) Checklist for the use of health apps [15],
6. Evaluating, but how? - Criteria for the evaluation and reporting of studies on health and medical apps [16],
7. How good are health apps? What determines quality & risk? What guidance is available? [17],
8. Mobile app rating scale: a new tool for assessing the quality of health mobile apps [18],
9. Methods box health literacy [19],
10. Health apps: significant lever for patient empowerment - potentials, but hardly used so far [20],
11. Professional checklist good health information [21].

Development of Evaluation Criteria

From the above publications, the evaluation criteria mentioned therein were entered into an Microsoft Excel file and assigned to overarching categories. In the next step, the discovered evaluation criteria were harmonized and evaluated. The evaluation criteria with the most matches were included in the list of evaluation criteria and categorized again. The preliminary evaluation criteria were presented and discussed in a focus group, consisting of the project team, health information experts, and public health experts. The evaluation criteria agreed upon in the focus group were prioritized within the focus group meeting and further refined in a quantitative survey of a total of 54 experts with different professions to filter out criteria considered less relevant and to make a final selection. Based on the results of the focus group and the survey, the evaluation criteria were revised and finalized and ended up in a long questionnaire, which is described in the results section.

Evaluation of theQuestionnaire With Selected Apps

The developed questionnaire was evaluated with apps from the following categories: (1) Nutrition, (2) Exercise, (3) Mental Health, and (4) Symptom Checker. These categories were chosen because they cover the main fields of healthy lifestyle. The selection of apps aimed to evaluate the most used apps in Austria in these categories. An attempt to select apps based on existing databases was unsuccessful because the data were at least one year old (eg, Statista [22]) or the databases used a distribution model that was not applicable in this context (eg, data.ai [23]).

For these reasons, an exploratory approach was chosen where 11 individuals from the project team, selected to ensure as much diversity as possible in terms of mobile phones, operating systems, and histories of installed apps, participated. This group consisted of 6 males and 5 females, with 6 participants using the Google Play Store and 5 using the Apple App Store. Each individual searched for the respective keywords in their personal app stores, ensuring a broad variety of search habits and preferences, and recorded the results—15 apps per category—in an Excel spreadsheet. The results were weighted using a pivot table after data cleaning (typos, adding app manufacturers). Then, it was ensured that the top 10 apps in each category were available in both app stores and could be used for evaluation.

The following inclusion and exclusion criteria were additionally considered in the app selection, as listed in Textbox 1:

Each app was assessed by 3 persons [AR, KE, SW] using the long questionnaire (see section results), whereby the persons cross-checked their results within each category of Apps.

Results

Overview

From the comprehensive evaluation criteria, we derived 2 questionnaires (short and long) to enable an informed and self-determined decision regarding the assessment of these apps. All questions in the following tables are to be answered with (yes/no) along with a detailed description. The responses serve as a guidance for the individual decision-making process regarding the app’s utility and quality.

Short Questionnaire

The short questionnaire summarizes the questions, which were most often answered with “no” in the evaluation of the long questionnaire and were rated as very important in the focus group and survey mentioned above. Table 1 shows the 6-items short questionnaire with an explanation of the reason for the importance as well as meaningful advice on how to self-assess the questions quickly.

App Selection

The selected apps to be tested against the long questionnaire is shown in Table 2. Detailed evaluation is not to be published due to possible violation of commercial law.

Evaluation Results

This chapter summarizes the evaluation results of the long questionnaire against the selected apps (Table 3).

User Survey

A quantitative survey with 414 participants was conducted to evaluate the value of health app features for achieving different goals from users’ perspectives, aiming for a balanced gender identity, age, and prior experience with health apps. The analysis focused on the relevance of goals such as support for physical activity (fitness), weight management, nutrition, mental health, and the monitoring of physical and mental symptoms. The findings showed that health apps in the “fitness/activity” and “symptom checker” categories were considered most important, while mental health apps were rated as less important. The ratings were influenced by gender, age, and the presence of chronic conditions, and prior experience with health apps also played a role. Women rated health app functions as more important than men, with a focus on supporting physical health and achieving weight management goals. Men prioritized physical activity support, while younger participants rated the importance of health app features higher than older ones, particularly for fitness-related goals. Individuals with chronic conditions placed more emphasis on functions promoting physical health and were more likely to pay for health apps.

Discussion

Principal Findings

Despite efforts in initiatives like Label2Enable [10], “So far, no unified concept has emerged to provide users with orientation regarding the quality of health apps. Different approaches with varying underlying criteria, often pursued by commercial entities, are being followed (eg, evaluation instruments for users, certifications, quality seals, test reports)” [12]

Our aim with this work has been to provide users with a unified concept for orientation regarding the quality of health apps, knowing well that the comprehensive instruments offered by Health technology assessments are not usable for users. The goal of this work was to create a questionnaire that could be used to evaluate 10 apps each in the areas of nutrition, exercise, mental health, and symptom checkers. The key points were summarized in the short questionnaire and supplemented with helpful tips and a “Meaningful advice” column, enabling users to assess the quality of an app in a short amount of time.

The user survey revealed that data privacy and security considerations played a minor role for most respondents, contrary to the questions outlined in the ”Short Questionnaire .” This could be due, on one hand, to the general perception of loss of control over one’s data, and on the other hand, the impacts of data privacy and security are not directly and immediately discernible to users. Data leaks with potentially negative consequences for individuals may only become known after a considerable period of time, and their effects may unfold gradually.

Additional education and information on “digital self-determination or self-defense” could help address this issue, enabling users to make clear and informed decisions about whether they prefer data-minimizing apps over those whose primary goal is to collect and market metadata. Simple questionnaires, as presented in this work, can raise awareness regarding data privacy and privacy, especially concerning sensitive health data, which may be perceived as critical in connection with initiatives like Elektronische Gesundheitsakte (ELGA) [25] but less so in the context of apps.

Digital health literacy already plays a role in selecting a suitable digital health applications, not only for users but also for therapists and medical professionals. Choosing a suitable, safe, and effective application from the multitude of digital health apps is a significant challenge. Currently, many users (35% in the user survey) select digital health applications based solely on star ratings in app stores, which have limited reliability [26]. In this context, the use of a questionnaire makes sense, especially regarding expected features and their evidence-based justification.

Apps are often downloaded based on advertising on social media, app stores, and recommendations from family and friends. This indicates that a quality-assured app recommended by a doctor could have significant potential for user adoption. Therefore, it is essential that reviews of these apps are quality-assured and reliable. This applies to both health apps and medical apps, which may target vulnerable groups such as people with chronic illnesses. Considerations regarding the German DiGAs [27] were taken into account throughout the project. Still, the outcome of this work is more geared towards private users rather than representatives of the health system. Nonetheless, we are pleased to note that Digitale Gesundheitsanwendungen (DiGAs - in English: digital health applications) have found a place in the preliminary version of the Austrian eHealth Strategy “M1.7 Creation of a unified process for the evaluation of Digital Health Applications and subsequently their introduction into standard care when demonstrably beneficial” [28]. This also applies to statements from the professional assessment of health apps. “The information offered is too extensive for laypeople”—here, too, a standardized evaluation and recommendation by authorities and health care providers would be advantageous.

In general, it can be noted that Austrian app developers adhere more frequently to regulatory requirements than international developers, since they most often met the evaluation criteria, especially in terms of data privacy and imprint. In this context, we would like to appeal to the responsibility of app providers to make a positive contribution to societal challenges A white paper in line with “Good App Development Practice” or regional requirements from pp stores could introduce even quality-assurance restrictions, particularly regarding regulatory frameworks (eg, explicitly demanding a legally compliant imprint) as collected in the results of this work.

Limitations

The selection of apps to be evaluated based on the established and prioritized criteria proved to be more challenging than initially anticipated. This was partly due to the lack of qualified sources for the previously desired criterion of “most commonly used apps in Austria,” requiring us to resort to an exploratory approach. Furthermore, the apps found as a result could not be fully tested and evaluated by all participants, as even during the implementation period, some apps were no longer available in the app stores for specific operating system versions. Nevertheless, a common denominator for joint evaluation had to be found.

In general, we want to emphasize that a framework for evaluating apps was established and applied, aiming to empower users to independently assess and evaluate both current and new apps.

We deliberately refrained from presenting individual apps or manufacturers in a positive or negative light.

Furthermore, we want to emphasize that our study has not been aimed at replacing the traditional instruments of Health Technology Assessment, as the evaluation criteria were created for the purpose mentioned above. However, this evaluation did not involve searching for clinical studies to demonstrate the effectiveness of the apps, and the apps were not assigned to risk classes according to Medical Device Regulation [29]. As such, for a comprehensive assessment of a health app, fulfilling the requirements of evidence-based medicine, further important domains such as clinical effectiveness, clinical safety, costs, organizational and ethical aspects need to be assessed.

Conclusions

In a rapidly changing landscape such as the app market, it is therefore even more important to have a solid, easy-to-use tool for evaluating health and medical apps at hand. We are aware that this tool may not be sufficient for evaluation by authorities; however, we would like to note that it can provide a good basis for it. We would like to support further research and projects in this direction.