Traditionally, multiple sclerosis (MS) has been categorized as having a relapsing-remitting or progressive course. However, recent work shed light on an underlying insidious progression, or a progression that is independent of relapses, even in patients previously thought to have a relapsing-remitting disease [1]. Therefore, minimizing or even eliminating progression is one of the goals of MS disease management. To achieve this goal, sensitive measures of MS-related functional ability are needed that can be frequently administered with minimal burden to the patient [2]. Here, digital health technology tools (DHTTs) such as smartphone sensor–based tests offer a new, promising strategy [3-5]. By taking advantage of the large variety of embedded sensors, smartphones enable the remote assessment of several functional domains affected by MS without supervision by a clinician [3]. However, user retention remains a barrier to the successful deployment of such DHTTs [6].

Assessments of user experience (UX) should, therefore, be considered when designing DHTTs [7]. They have attracted increasing attention, as they provide important insights to enhance usability, engagement, perception, and satisfaction [8-11]. With different UX designs available, it is important to keep in mind that their effectiveness depends on both the intended users and the environment in which they are being used. Hence, a comprehensive assessment of UX should include the study of both qualitative and quantitative UX in different environments for an extended period. The primary focus of qualitative UX assessments is to characterize the individual user’s experiences with the DHHT and to elucidate fine-grained aspects of why and how users engage with it [12-14]. Quantitative UX assessments, by comparison, provide strong indicators of the duration and frequency of user engagement and retention and are typically conducted in larger cohorts.

Here, we present a qualitative and quantitative UX analysis of the Floodlight Open app (Figure 1). The app comprises patient-reported outcomes and smartphone sensor–based tests that assess mood, cognition, hand-motor function, postural stability and gait, and mobility levels [15]. It was deployed in Floodlight Open, a global, open-access, digital-only study that was designed to understand the drivers and barriers in the deployment of the app in a naturalistic setting without supervision and onboarding by a clinician, in a broad, multinational study cohort [15]. The UX analyses presented here allow us to gain a better understanding of the participants’ behavior with the Floodlight Open app. This will help make informed UX design decisions to improve engagement with the Floodlight technology. Qualitative UX is assessed through semistructured interviews conducted with a subset of US participants with MS [16] to identify important elements that could improve UX and user engagement. In addition, quantitative UX is assessed by evaluating the impact of UX design changes, which were motivated by the learnings from the qualitative UX analysis, on user engagement and retention using data collected from Floodlight Open.

Floodlight Open’s full study design and inclusion and exclusion criteria have been previously reported [15]. Adults with or without MS who owned a suitable iOS (Apple Inc) or Android (Google LLC) smartphone and resided in 1 of the 17 participating countries were permitted to take part. The MS status was self-declared by the participants. To join Floodlight Open, participants first registered on the Floodlight Open web portal and completed the informed e-consent process. After successfully completing these 2 steps, each participant received their personal Floodlight Open unique identifier (FLO ID) and an activation code (ie, an activation token [15]) via email. Both were required to activate the Floodlight Open app on the participant’s personal smartphone device. The email also provided a link to the Floodlight Open web portal, where participants could request a new activation code (the activation code was valid for 48 hours, after which it expired), access their account to update their profile, view their results, or withdraw from the study.

After the activation of the Floodlight Open app, several tests, or assessments, were made available to the participants. These included a patient-reported outcome assessing the participants’ mood (Daily Mood Questionnaire [DMQ] [15]) and a series of smartphone sensor–based active tests (ie, tests that require active input from the user) measuring cognition (Information Processing Speed [IPS] [15] and IPS Digit–Digit [15]); hand-motor function (Pinching Test [PT] [15,17] and Draw a Shape Test [DaS] [15]); and postural stability and gait (Static Balance Test [SBT] [15], U-Turn Test [UTT] [15,18], and Two-Minute Walk Test [2MWT] [15]; Figure 2).

In addition, data on mobility level were passively collected without requiring input from the user (passive monitoring) through life-space measurement [15], which measured the distance between the 2 farthest GPS coordinates detected during the day that were at least 500 m apart, resulting in 1 measurement per day. Since this passive monitoring does not require any active input from the user, it is not included in our UX analyses.

The DMQ and all active tests except the 2MWT were administered in a predefined, fixed sequence comprising the daily activities. The DMQ was administered first, followed by the IPS, IPS Digit–Digit, PT, DaS, SBT, and UTT. The 2MWT was administered separately from these daily activities, as this test required the participant to walk in a straight line for 2 minutes, which may not always be possible at the time of taking the other active tests. The tests could be taken up to once a week (IPS) or once a day (all other tests), but the actual test frequency was not systematically enforced. For example, the UTT, SBT, and 2MWT could be skipped if participants determined that the conditions for safe execution, including environmental and physical factors, could not be met that day.

Data collected with the Floodlight Open app were encrypted and electronically stored on 2 specific, secure cloud databases, which were made publicly available for the duration of the study and maintained by the study initiator. To ensure confidentiality, all participant information was pseudonymized through association with the personal FLO ID. Furthermore, no personal identifiable information was collected while the participants executed the Floodlight Open tests. This meant that GPS coordinates were obfuscated and excluded from the public data set to protect participants’ privacy.

The protocol, informed e-consent forms, and relevant supporting information were reviewed and approved by the appointed central institutional review boards or ethics committees before the study was initiated in each participating country, as applicable, in accordance with each country’s regulatory requirements [15]. The institutional review board for the United States was the Western Institutional Review Board in Puyallup, Washington (approval: 20180617).

The qualitative UX substudy was conducted by a health care delivery network (Sutter Health) in a subset of US participants with MS who took part in Floodlight Open and agreed to download the app and use it daily for 30 days [16]. This substudy aimed at elucidating finer-grained aspects of why and how users engage with the Floodlight Open app in a particular manner and at providing information on UX design features that could improve UX and user engagement. Individual, semistructured interviews were conducted to gain insights into the participants’ experiences with the Floodlight Open app and the perceived benefits thereof (Table 1). The use of individual interviews offered privacy and enabled the exploration of each participant’s interaction with the app. A multifaceted recruitment strategy was applied to involve participants of a broad age range to combine the experiences of the tech-savvy younger generation with that of the older generation. Most participants were using their smartphone and their computer daily. No preferential sampling of participants with either negative or positive experiences was applied. Any duration of use of the Floodlight Open app was of value because discontinued use and negative experiences can provide valuable insights and complement data from persistent participants.

Semistructured interviews were conducted via phone by a trained interviewer, with 2 attempts made to reach each participant. The interviews lasted between 15 and 20 minutes and were audiotaped and fully transcribed for further analysis. A reductionist approach was applied to deconstruct implicit and explicit responses into manageable variables. The qualitative paradigm was crucial to appreciating, observing, and deducing participants’ experiences [19]. Through an inductive analysis of the data collected, the common needs of the participants were identified, which should be considered to improve the UX and to develop preliminary conclusions on the need for specific UX features.

Until November 2, 2021, when the study closed in the United States, the Floodlight Open app was downloaded 5225 times, including 4240 (81.15%) times on iOS devices and 985 (18.85%) times on Android devices, across the 17 participating countries. The baseline demographics of the US participants with MS included in the qualitative and quantitative UX analyses on reminders and notifications are presented in Table 3.

A total of 22 US participants were enrolled in the qualitative substudy, of which 15 (68%) were interviewed. Between 7 and 19 participants are considered adequate for qualitative research [30].

The qualitative UX analysis reveals key points for improving the UX with the Floodlight Open app (Table 4). While many participants did not face any difficulties downloading the Floodlight Open app and were willing to perform the tests daily, some were unaware or confused about the activation process despite instructions being provided on the study’s web portal (see Need for a More Seamless User Journey section in Table 4). Other participants reported that they did not complete the activation process due to distractions or the lack of reminders to complete this step. Here, a more seamless journey from registering on the web portal to activating the Floodlight Open app could improve the UX. In addition, participants who successfully activated the Floodlight Open app self-reported that they used it for 2 to 30 days and agreed that screen contents, prompts, and information were easy to see and understand. However, they also reported that they often forgot to perform the tests or felt it was not always feasible to perform all the tests in one setting. Therefore, configurable reminders and push notifications would have been beneficial (see bottom section in Table 4). Such UX design features could be used to set up reminders to perform the tests or to stop the current test and plan to complete the tests later. Furthermore, participants were not always available to perform tests that required them to stand or walk:

Consequently, in the subsequent section on quantitative UX analysis, UX with the Floodlight Open app was investigated with respect to DA, DA_x, and 2MWT.

Smartphones enable out-of-clinic assessments of chronic neurological diseases. Despite the rapidly increasing number of mobile health care apps available for consumers’ self-care, there is a paucity of research into the UX of DHTTs for MS. In this paper, we present our qualitative and quantitative UX analyses of the Floodlight Open app and demonstrate that the adoption of key UX design features markedly improved user acquisition, engagement, and retention. Understanding how the presence or absence of specific UX design features affects participants’ experiences offers important guidance for the refinement of an existing or the design of a new DHTT. Therefore, UX design features are a significant consideration when designing and evaluating such tools.

Assessing MS symptoms over time with the Floodlight Open app requires specific user stimulation and guidance through seamless and flexible user journeys enhanced by engaging features to improve the overall UX. By considering participants’ feedback collected in the quantitative UX substudy, we enhanced the onboarding process by implementing a more seamless user registration and app activation journey and improved the UX through configurable reminders and push notifications. We derived several distributions from real-world data that describe key aspects of participants’ behavior in their environment. Such measurements and their statistical analysis might be used to help describe realistic user behavior and design health care apps similar to our Floodlight Open app. Our findings allow us to gain a better understanding of users’ experience in such settings and to make informed design decisions for self-monitoring mobile apps that support people with MS.

Users’ first experience with the registration to the Floodlight Open study and the activation of the study app is important to increase user acquisition. Furthermore, the first few days after activating the app are similarly critical for long-term engagement and retention. For instance, user acquisition, engagement, and retention can all be improved by guiding users throughout the Floodlight Open study registration, app activation process, and daily activities, reminding them about the daily activities to be performed and allowing them to plan when to perform the daily activities. Our results demonstrate that a seamless UX is essential to minimize the rate at which users drop out along the end-to-end journey. With a seamless user journey, users are led along the registration-activation journey and are not distracted by unrelated content or activities, resulting in a higher conversion rate from creating an account to activating the Floodlight Open app.

Configurable reminders and push notifications can help keep the participants engaged with mobile health care apps, such as the Floodlight Open app, and help retain them beyond the initial onboarding phase. Our findings show that continuous use times, test completion rates, and user retention rates are generally higher with configurable reminders and push notifications than without configurable reminders and push notifications, whereas inactivity time is comparable between the 2 cohorts. This improvement in user engagement is most evident for 2MWT from the first day and for DA and DA_x after the initial onboarding phase, during which the participants are still exploring the Floodlight Open app. Therefore, to fully optimize the UX, the app should ask the users to accept and receive reminders and notifications only after this initial onboarding phase [32]. Interrupting users with notifications too soon after the activation of the app may create frustration and may ultimately lead them to abandon the app. The app should, therefore, minimize the mental and physical interaction efforts during the first few days of use.

Comparing the different test series, we noted slightly worse user retention for 2MWT than for either DA or DA_x. Taking the 2MWT requires the greatest effort, which is in line with some dissatisfaction reported in a previous proof-of-concept study [33]. It is possible that the study participants did want to take the 2MWT but did not find a suitable opportunity to take it each day. In fact, participants in Floodlight Open were found to persistently take the 2MWT at least once per week [15], which suggests that a more flexible or less frequent assessment schedule could be beneficial.

Although several other smartphone apps are available for people with MS (for a review on MS apps, refer to the study by Howard et al [34]), to our knowledge, this is the first analysis to assess the impact of UX design features on user engagement and retention. Nonetheless, previous analyses have shown that user engagement and retention remain two of the barriers to the long-term successful deployment of DHTTs (for reviews, refer to works of Pratap et al [6] and Amagai et al [35]). Several UX design features have been suggested to improve the overall UX, including a more seamless user journey, configurable reminders, and push notifications [23,35-38]. However, differences in the definitions used to assess user retention (eg, user retention definitions based on the user simply opening or interacting with the DHHT [31] vs completing a test or series of tests [6]), limited statistical power due to small sample sizes [37], and differences in the observed study period [37] may make direct comparisons of user retention across different DHHTs challenging.

To compare like with like, we computed the user retention with the Floodlight Open app for the same observation period as that reported in the study by Rosenfelder [31] for medical as well as health and lifestyle apps. Both data sets indicate that the largest reduction of users occurs from days 1 to 3, whereas the reduction in user retention is low between days 7 and 30. This suggests that a significant proportion of users who will not use the Floodlight Open app (or other health care apps) in the long term will stop using the app within the first 3 days and that this dropout is likely to be observed within the first 7 days. These results also suggest that participants who use the app for >3 days are likely to use it for at least 30 days. A similar dropout during the first 7 days has also been previously reported for the whole study cohort of Floodlight Open [15].

Of note, our user retention rates are approximately 2- to 6-fold higher than those reported by Rosenfelder [31] (Table 5) despite using a more conservative definition of user retention (completing a test series vs simply interacting with or opening the app). It is conceivable that not all apps included in the report by Rosenfelder [31] feature the same UX design features as the Floodlight Open app does. However, favorable user retention rates were observed with the Floodlight Open app even when the app did not have configurable reminders or push notifications. This suggests that differences in the perceived benefit of the app might have also contributed to the favorable retention profile, although other reasons cannot be excluded [39]. Despite our favorable user retention and user engagement findings, a previous analysis showed that improvement in smartphone sensor data collection could be achieved through passive data collection methods [15].

A few limitations are noted. First, while participants could turn off the configurable reminders and push notifications, this action was not logged. Instead, we compared participants who enrolled before with those who enrolled after these features were implemented. For the former cohort, we included only the data collected up to the implementation of reminders and notifications in our analyses to be able to assess the impact of reminders and notifications. Second, our results were influenced by the duration of the study and by incoming participants who activated the Floodlight Open app on different dates over the duration of the study and might not be part of the study for the same number of days. Besides, for events approaching the duration of the study, there is an artificially lower likelihood of observation, and events lasting longer than the study cannot be observed. Finally, the use of the Floodlight Open app in a clinical study other than the Floodlight Open observational study may boost user engagement and retention due to additional motivational incentives. Such clinical referrals have been shown to improve user retention [6]. However, information on which participants participated in such studies is not available, and without it, accounting for it is not possible.

We presented the qualitative and quantitative UX analyses of the Floodlight Open app, a DHTT for MS. Learnings from the qualitative UX analysis led to the implementation of a more seamless user journey and the introduction of configurable reminders and push notifications. These US design features improved user acquisition, user engagement, and user retention. The continued assessment of UX and improvement of UX design features are critical steps in optimizing the long-term adoption of the Floodlight Open technology and similar DHTTs.