The general implementation and transcription procedures of the interviews have been described previously [27]. In short, an interview guideline based on the UTAUT model was developed. After an introduction including verbal consent, some basic demographic questions followed. Participants were asked about each app separately. First, participants were asked to describe their general experiences with the app, followed by more detailed questions to cover all aspects of UTAUT. Participants in the FS were contacted during the study to assess their willingness to take part in an interview. Interviews were conducted between October 2020 and June 2021, mostly via videoconference. Written consent to participate in the interviews was obtained beforehand. Interviews were conducted in the first language of the participants and recorded with their consent. They were transcribed verbatim or pseudonymized by a transcription service. Interviews in Dutch were translated into English with an online translation tool and double-checked by a native speaker. Quotations used in this paper were slightly edited for readability without altering their meaning. All interviews were analyzed using content analysis following Mayring [30] with MAXQDA 2022 (version 22.8.0; VERBI GmbH). Overall, 59 complete interviews and 2 summaries of interviews (where recording had not worked) were included in the analysis. Initially, 4 broad top-level codes were defined to classify the interview content according to the 4 constructs of technology acceptance. In the next step, potential subcodes were delineated on the basis of the interview guide. Subsequently, the interviews were analyzed using these preexisting codes, with the additional step of using an inductive method to derive new codes. Applying this method to both apps resulted in comparable coding systems that still account for individual differences. The resulting code system is illustrated in Figure 2.

While the UTAUT model was used as a foundation for both the interview guideline and the analysis, it was slightly adapted to fit this study. As participants in a medical study, users did not expect the devices to be personally useful and were unable to judge their usefulness to the study team. In addition, as we interviewed participants after their use of the device, “expectation” seemed to be an inaccurate term in this case. Therefore, we decided to use the terms “performance” and “effort or ease of use” instead of performance expectancy and effort expectancy, respectively.

Interviews were conducted separately for each participant group until data saturation was reached. To ensure a diverse sample, we conducted at least 1 interview per patient group and research center. When new interviews in a patient group did not lead to new valuable insights, we assumed data saturation. This approach led to different sample sizes. A total of 61 interviews were conducted. As only 1 patient with systemic lupus erythematosus could be interviewed and therefore data saturation is not clear, this interview was excluded from analyses.

Ethics approval was granted by the ethics committee of Kiel University, Germany (Ethikkommission der Medizinischen Fakultät der CAU zu Kiel, approval D491/20) and the Health Research Authority and Health and Care Research Wales (IRAS project ID 282329, REC reference 20/PR/0185). Participants gave written and verbal informed consent before taking part in the study. Interviews were voluntary for participants of the larger FS. To ensure participant privacy, participants were assigned a random code. The code key is stored, encrypted, and password-protected. Interview transcripts were pseudonymized. Voice and video recordings are stored separately in accordance with the General Data Protection Regulation.

Table 1 provides an overview of the number of interviews per group and site, while Table 2 shows the demographic characteristics of the sample.

Figure 2 shows the general code system used for both apps. Some of the second- and third-level codes were only used for one of the apps. Quality of the questions, quality of speech function and tests, as well as usability of the reminder function and reactions of peers were only needed for the symptom tracking app, while feedback was only used for the cognitive test app.

This study considers the technology acceptance of 2 different apps in the context of the IDEA-FAST FS. We interviewed participants who used a symptom and activity app as well as a cognitive test app over 4 weeks. Using the UTAUT [13] model, we identified different themes of technology acceptance in the use of these apps.

For the users, performance played a less crucial role in technology acceptance, whereas the aspect of effort gained importance. Both facilitating conditions as well as social influence seemed to play minor roles in the acceptance of the apps.

When users expect an mHealth app to be helpful to them, they show more willingness to use it [1,6,16]. This perceived usefulness has been shown to be a good predictor for the intention to use mHealth apps [17,18,21,23]. In this study, the apps did not provide an apparent impact on the participants’ health. Therefore, motivation and adherence to the schedule were issues with both apps. Some participants did not see value in answering the questions of the symptom tracking app and saw it as a burden, while others enjoyed the opportunity to track their behavior and symptoms. The cognitive test app varied in popularity among participants. Some users found the PVT boring, but it was well liked by most, as was the DSST. However, many participants disliked the NBX, found it too hard or frustrating. This led to compliance problems, such as randomly clicking through the test, ignoring or missing scheduled sessions, or ceasing to answer it altogether. A possible reason for this could be that the task was developed for healthy users. The cognitive impairments of participants with Parkinson disease and Huntington disease, along with the relatively high average age of the sample, contributed to the task being perceived as too difficult. As a result, the task was adjusted during the FS. However, since the adjusted NBX was still perceived as too difficult, it was not included in the subsequent clinical observation study.

An important aspect of motivation for participants seemed to be feedback on their data. While not many participants reported interest in the results from the symptom tracking app, those who enjoyed the app used it as a tool to track their behavior and symptoms, especially fatigue. Possibly, these were patients who were concerned about their fatigue before joining the study. In contrast, many participants voiced a wish for feedback from the cognitive tests. Feedback could help increase motivation if participants were able to see their performance over time. If an mHealth app itself does not seem inherently helpful to the user, feedback might serve as a useful way to engage them. At the same time, feedback could be considered as an intervention, which researchers might wish to avoid. Feedback can be a helpful or interesting resource users can gain from their app use.

People tend to use mHealth apps they perceive as easy to understand and to use. This effect has been shown in previous studies [1,6,16,18] and can be observed here as well. While users agreed that the apps did not take too much time, the frequency of use required by the study design was more problematic. Especially having to use the apps at specific time points was seen as disruptive and led to missing data. The 2 apps described in this study had to be used several times per day. Even though the sessions were short, these numerous sessions meant a considerable amount of extra effort, especially since the participants had to adhere to certain time constraints. It was not surprising that almost all participants reported they had not been able to attend all sessions, especially those who were more active or working. Participants suggested adjustable time frames as a solution to account for differences in daily routine or to find individual times they would be able to handle the apps, which should be considered by the clinical team to find a compromise between the patient’s needs and study goals. When choosing an app for a scientific study, it is beneficial if it is available for all common smartphone operating systems. Often, participants prefer using their own phone rather than a study phone. Study phones constitute an additional device participants have to familiarize themselves with, charge regularly, carry around, and keep in mind to use. They tend to be forgotten or ignored more easily.

Both social influence and facilitating conditions did not seem to have significant impact on the acceptance of the apps in this study. An explanation could be the study context. As participants were using the apps without expected impact on their health, it is not surprising that family members and peers did not have specific expectations or motivating sentiments regarding app use. Therefore, opinions of peers seem to have had only a slight impact on their willingness to use the apps. This finding is not entirely inconsistent with the literature, as some studies have found no influence of social influence on the intention to use a technology [20,31]. Other studies have shown that opinions of peers influence the decision to use an mHealth app [21,23]. In the context of study-related apps, social influence may affect the decision to participate in the study rather than the use of the app itself. Another interesting finding was that participants did not voice major concerns regarding data privacy. Data privacy and security have previously been reported as a valid concern for users of mHealth apps [10,16,25,32], though others did not find that privacy concerns predicted intention to use mHealth apps [17,18,21]. As the symptom tracking app gathers quite personal information, concerns regarding data privacy had to be expected. When asked about data privacy, many participants cited trust in the study team and medical data security standards as reasons to feel safe.

Regarding facilitating conditions, salient factors such as costs for the use of mHealth apps or availability [16,23,33] were not an issue in this case. Both apps as well as study smartphones were provided for the participants. It seems that the provided support in the form of instructions and the study team was sufficient for most participants to feel confident about the correct use of the apps. There were some uncertainties concerning the symptom tracking app, like not knowing what to report using the speech function and not knowing how much time there was for answering the questions. Facilitating conditions have been found to be very influential on the intention of using a technology in several studies [1,33], while others did not find a connection [6]. It seems probable that due to the high amount of support and lack of limiting factors, the positive impact of facilitating conditions on the use of the apps was present. This assumption is supported by the fact that many participants stressed the good support during the study and voiced their appreciation for the study team.

Some aspects of these apps were controversial among participants. While many participants disliked the NBX of the cognitive test app, some voiced their appreciation for feeling challenged by the task. Many participants described the frequent documentation of their well-being as tedious, although some appreciated the possibility to track their symptoms. While studying the opinions and experiences of users is a valuable tool to ensure acceptance, users have different desires and needs. In choosing suitable mHealth apps, one size will most certainly not fit all. While some aspects of these apps are necessary for study success, such as the documentation in the symptom tracking app, others might be customizable. An example is the reminder function of the symptom tracking app, where users can decide for themselves if they want to be reminded or not.

Limitations and strengths of this study align with those previously described [27]. In short, due to the design of the study, participants used multiple devices as well as the mHealth apps at the same time, which might have influenced their opinions. Due to the different number of interviews in each group, disease-specific aspects might not have been entirely captured. In addition, the COVID-19 pandemic might have had an influence on the acceptance of the 2 apps, since participants were partly living in lockdown or had altered their normal routines. We are also aware that these qualitative interviews primarily reflect app modifications made specifically for this FS, including support provided by the clinical team. Thus, the findings cannot be generalized to the assessment of app use in different contexts. The results represent the experiences of the FS participants with the apps implemented in a specific study with its own design choices. The requirements of apps can be different, depending on their context of use.

This study represents the opinions and voices of a large and diverse sample on the acceptance of 2 mHealth apps with different aims and uses. This enables us to draw conclusions, which might also be useful for other studies planning to use mHealth apps, as well as help in the development of new apps. Conducting these interviews during the duration of the FS helped us to improve and select suitable apps for use in the clinical observation study. As described earlier, the NBX of the cognitive test app was adjusted to fit better to the study population. Its use was discontinued for the clinical observation study. Our experience underlines the importance of involving users and patients in the development and design of mHealth apps in order to achieve a user-friendly and meaningful result [11].

The choice of appropriate mHealth apps for a research project can be a challenging decision to make. The efficacy of any data collection app is contingent upon the extent to which its users actually use it. Consequently, when developing or selecting an mHealth app for a research project, it is essential to consider technology acceptance. The effort expended by the user should be considered as an integral component of the evaluation process. The ideal app is one that is straightforward and intuitive to use [24]. Patients will have divergent preferences regarding the impact of the study on their lives. Consequently, the integration of customizable features, where feasible, may enhance technology acceptance. These may encompass adjustable reminder functions, time points, and time frames for use of the app. The minimization of user effort is also relevant to the decision regarding the use of study phones. Many patients may prefer their personal smartphones as opposed to designated study phones. Therefore, ensuring that the app is available for all common smartphone operating systems might be beneficial [23]. The necessity of carrying 2 phones is thus eliminated, and the potential for patients to forget the phone at home and miss vital data is also minimized. Evidently, the time expended in using the app constitutes a significant proportion of the overall effort involved. Most cognitive testing and symptom tracking apps require patients to use them for a fixed period of time. Should this be required on a daily basis, or several times per day at designated times, it would necessitate a significant degree of effort. To minimize effort, it is vital to keep the required app time as short and infrequent as possible.

Considering the role of performance in technology acceptance, the pivotal question pertains to the extent to which the technology benefits the patient. The use of cognitive testing and symptom tracking apps within the context of research studies may not necessarily result in tangible health benefits for the patient. This is particularly relevant in cases where apps are used over an extended period of time. In such cases, consideration should be given to alternative factors of motivation. These elements may include gamification mechanisms or feedback on data in noninterventional studies. In the context of cognitive testing apps, the motivational factors of enjoyment and challenge can be leveraged. A task that is perceived as too easy may quickly become tedious to the user, while those that are too challenging may result in demotivation over time. The process of identifying an appropriate level of difficulty for the target population can be a difficult yet rewarding endeavor.

The role of social influence in the technology acceptance of mHealth apps in the context of medical studies may be less significant. It is possible that the opinion of significant others may influence patients’ decisions regarding participation in the study, thereby reducing the impact of social influence during the actual use of the apps. Data security could be a cause of concern for patients. The provision of information regarding data security measures during the recruitment process is therefore a valuable use of time [25].

Some salient factors that are typically considered facilitating conditions of technology acceptance, such as costs or availability [23], are typically negligible in the context of medical studies. Nevertheless, the pivotal role that technical and organizational support play in facilitating patient compliance has to be taken into account. The provision of adequate instructions and support should be considered a prerequisite [21]. The establishment of positive relations between patients and the study team can be a significant motivating factor, as well as enhancing satisfaction with the study itself.

This study describes the technology acceptance of 2 mHealth apps. The primary objective of this study was to assess the usability of these apps in a follow-up investigation. As part of a larger evaluation study, potential issues were identified. They include demotivation due to the level of difficulty experienced with one of the cognitive tasks and challenges with adhering to the stipulated time schedule for the symptom tracking app. These issues were addressed through a variety of approaches. For instance, the challenging task was set aside for use in the follow-up. To support adherence, time points were limited to those essential for the study’s success while allowing as much flexibility as possible. The second objective of this study was to assess the specific challenges researchers encounter when selecting suitable mHealth apps for their own research, and what characteristics an app must possess to be considered a useful tool for health research. The incorporation of patient perspectives in the device selection process has been demonstrated to be beneficial and valuable. The findings of this study provide a valuable contribution to the field of medical research by offering insights that may inform the selection of appropriate mHealth apps. The study emphasizes the importance of effective communication and testing of all materials used in conjunction with patients and intended users prior to the commencement of research studies.