User Experience of a Large-Scale Smartphone-Based Observational Study in Multiple Sclerosis: Global, Open-Access, Digital-Only Study

Introduction

Background

Traditionally, multiple sclerosis (MS) has been categorized as having a relapsing-remitting or progressive course. However, recent work shed light on an underlying insidious progression, or a progression that is independent of relapses, even in patients previously thought to have a relapsing-remitting disease [1]. Therefore, minimizing or even eliminating progression is one of the goals of MS disease management. To achieve this goal, sensitive measures of MS-related functional ability are needed that can be frequently administered with minimal burden to the patient [2]. Here, digital health technology tools (DHTTs) such as smartphone sensor–based tests offer a new, promising strategy [3-5]. By taking advantage of the large variety of embedded sensors, smartphones enable the remote assessment of several functional domains affected by MS without supervision by a clinician [3]. However, user retention remains a barrier to the successful deployment of such DHTTs [6].

Assessments of user experience (UX) should, therefore, be considered when designing DHTTs [7]. They have attracted increasing attention, as they provide important insights to enhance usability, engagement, perception, and satisfaction [8-11]. With different UX designs available, it is important to keep in mind that their effectiveness depends on both the intended users and the environment in which they are being used. Hence, a comprehensive assessment of UX should include the study of both qualitative and quantitative UX in different environments for an extended period. The primary focus of qualitative UX assessments is to characterize the individual user’s experiences with the DHHT and to elucidate fine-grained aspects of why and how users engage with it [12-14]. Quantitative UX assessments, by comparison, provide strong indicators of the duration and frequency of user engagement and retention and are typically conducted in larger cohorts.

Aims

Here, we present a qualitative and quantitative UX analysis of the Floodlight Open app (Figure 1). The app comprises patient-reported outcomes and smartphone sensor–based tests that assess mood, cognition, hand-motor function, postural stability and gait, and mobility levels [15]. It was deployed in Floodlight Open, a global, open-access, digital-only study that was designed to understand the drivers and barriers in the deployment of the app in a naturalistic setting without supervision and onboarding by a clinician, in a broad, multinational study cohort [15]. The UX analyses presented here allow us to gain a better understanding of the participants’ behavior with the Floodlight Open app. This will help make informed UX design decisions to improve engagement with the Floodlight technology. Qualitative UX is assessed through semistructured interviews conducted with a subset of US participants with MS [16] to identify important elements that could improve UX and user engagement. In addition, quantitative UX is assessed by evaluating the impact of UX design changes, which were motivated by the learnings from the qualitative UX analysis, on user engagement and retention using data collected from Floodlight Open.

‎

Methods

Participants and Study Design

Floodlight Open’s full study design and inclusion and exclusion criteria have been previously reported [15]. Adults with or without MS who owned a suitable iOS (Apple Inc) or Android (Google LLC) smartphone and resided in 1 of the 17 participating countries were permitted to take part. The MS status was self-declared by the participants. To join Floodlight Open, participants first registered on the Floodlight Open web portal and completed the informed e-consent process. After successfully completing these 2 steps, each participant received their personal Floodlight Open unique identifier (FLO ID) and an activation code (ie, an activation token [15]) via email. Both were required to activate the Floodlight Open app on the participant’s personal smartphone device. The email also provided a link to the Floodlight Open web portal, where participants could request a new activation code (the activation code was valid for 48 hours, after which it expired), access their account to update their profile, view their results, or withdraw from the study.

After the activation of the Floodlight Open app, several tests, or assessments, were made available to the participants. These included a patient-reported outcome assessing the participants’ mood (Daily Mood Questionnaire [DMQ] [15]) and a series of smartphone sensor–based active tests (ie, tests that require active input from the user) measuring cognition (Information Processing Speed [IPS] [15] and IPS Digit–Digit [15]); hand-motor function (Pinching Test [PT] [15,17] and Draw a Shape Test [DaS] [15]); and postural stability and gait (Static Balance Test [SBT] [15], U-Turn Test [UTT] [15,18], and Two-Minute Walk Test [2MWT] [15]; Figure 2).

‎

In addition, data on mobility level were passively collected without requiring input from the user (passive monitoring) through life-space measurement [15], which measured the distance between the 2 farthest GPS coordinates detected during the day that were at least 500 m apart, resulting in 1 measurement per day. Since this passive monitoring does not require any active input from the user, it is not included in our UX analyses.

The DMQ and all active tests except the 2MWT were administered in a predefined, fixed sequence comprising the daily activities. The DMQ was administered first, followed by the IPS, IPS Digit–Digit, PT, DaS, SBT, and UTT. The 2MWT was administered separately from these daily activities, as this test required the participant to walk in a straight line for 2 minutes, which may not always be possible at the time of taking the other active tests. The tests could be taken up to once a week (IPS) or once a day (all other tests), but the actual test frequency was not systematically enforced. For example, the UTT, SBT, and 2MWT could be skipped if participants determined that the conditions for safe execution, including environmental and physical factors, could not be met that day.

Ethical Considerations

Data collected with the Floodlight Open app were encrypted and electronically stored on 2 specific, secure cloud databases, which were made publicly available for the duration of the study and maintained by the study initiator. To ensure confidentiality, all participant information was pseudonymized through association with the personal FLO ID. Furthermore, no personal identifiable information was collected while the participants executed the Floodlight Open tests. This meant that GPS coordinates were obfuscated and excluded from the public data set to protect participants’ privacy.

The protocol, informed e-consent forms, and relevant supporting information were reviewed and approved by the appointed central institutional review boards or ethics committees before the study was initiated in each participating country, as applicable, in accordance with each country’s regulatory requirements [15]. The institutional review board for the United States was the Western Institutional Review Board in Puyallup, Washington (approval: 20180617).

Qualitative UX Analysis

The qualitative UX substudy was conducted by a health care delivery network (Sutter Health) in a subset of US participants with MS who took part in Floodlight Open and agreed to download the app and use it daily for 30 days [16]. This substudy aimed at elucidating finer-grained aspects of why and how users engage with the Floodlight Open app in a particular manner and at providing information on UX design features that could improve UX and user engagement. Individual, semistructured interviews were conducted to gain insights into the participants’ experiences with the Floodlight Open app and the perceived benefits thereof (Table 1). The use of individual interviews offered privacy and enabled the exploration of each participant’s interaction with the app. A multifaceted recruitment strategy was applied to involve participants of a broad age range to combine the experiences of the tech-savvy younger generation with that of the older generation. Most participants were using their smartphone and their computer daily. No preferential sampling of participants with either negative or positive experiences was applied. Any duration of use of the Floodlight Open app was of value because discontinued use and negative experiences can provide valuable insights and complement data from persistent participants.

Semistructured interviews were conducted via phone by a trained interviewer, with 2 attempts made to reach each participant. The interviews lasted between 15 and 20 minutes and were audiotaped and fully transcribed for further analysis. A reductionist approach was applied to deconstruct implicit and explicit responses into manageable variables. The qualitative paradigm was crucial to appreciating, observing, and deducing participants’ experiences [19]. Through an inductive analysis of the data collected, the common needs of the participants were identified, which should be considered to improve the UX and to develop preliminary conclusions on the need for specific UX features.

Quantitative UX Analysis

Participants

The quantitative UX analysis was conducted in all US participants who enrolled in Floodlight Open [15]. The analysis was restricted to US participants for several reasons. First, information on user acquisition, including the date and time of activation of the FLO ID, was collected only in the United States. Second, the release dates of the iOS and Android versions of the Floodlight Open app differed for each country [15]. Due to these different release dates, only the United States offered a sufficiently long period during which the app’s use without the new UX design features could be studied. Finally, limiting the analysis to US participants allowed us to keep the quantitative UX analysis as consistent as possible with the qualitative UX analysis, which was conducted in a subset of the US participants, thus reducing the impact of differences in demographics and user behavior between the 2 analyses.

Implementation of New UX Design Features

The quantitative UX analysis was conducted to investigate the impact of the new UX design features identified in the qualitative UX analysis on user engagement and retention. One of the new UX design features identified in the qualitative UX analysis was a more seamless user journey, which guides users from registration to the activation of the Floodlight Open app (Figure 3). The original user journey involved registering on the Floodlight Open web portal; downloading the Floodlight Open app from the local app store; and activating the app by entering the FLO ID and activation code, which the participants received in a notification email. This last step required participants to switch multiple times between the notification email and the Floodlight Open app to manually enter their FLO ID and activation code (original user journey in Figure 3). These complex interactions with multiple user interfaces may frustrate participants, who may eventually abandon the activation process. Therefore, to minimize nuisance factors and reduce user loss, a more seamless user registration and app activation journey was implemented by deep linking the notification email with the Floodlight Open app. This allowed participants to launch the Floodlight Open app directly from the notification email, if already installed on their phone, or to download it from their local app store, with the fields for the FLO ID and activation code already filled out (ie, deep linked). Thus, this more seamless user journey eliminated the requirement of searching for the FLO ID and activation code in the notification email and manually inserting them in the right fields in the Floodlight Open app to activate it (more seamless user journey in Figure 3).

‎

In addition, configurable reminders and push notifications were introduced as the second new UX design feature (Table 2). Configurable reminders provided participants the option to set up reminders to perform the tests. Participants could set up daily or weekly reminders by tapping on the reminder icon, which was visible on the app’s dashboard (Figure 2). Push notifications, by comparison, were designed to serve two purposes: (1) to encourage inactive participants to reengage with the Floodlight Open app, for example, when they closed the app and did not complete the tests or have been inactive for some time, and (2) to motivate committed users to keep performing the tests. Tapping on the notification took the participants directly to the right entry point of the Floodlight Open app to perform the uncompleted or next test. To enable this new feature, participants had to grant permission to receive push notifications.

The push notifications were designed as a friendly communication with a strong emphasis on the emotional context (Table 2), as the emotional interpretation of a message can impact the user’s response [20,21]. The emotional interpretation begins with the perception of a friendly communication and leads to the planning and execution of a responsive action. Notifications designed in this way are less likely to be ignored and can help create the perception of a human touch within the Floodlight Open app. Hence, they are quite effective for eliciting short-term actions [22] and are an important variable in improving adherence [23].

The frequency and logic with which push notifications were triggered were carefully considered to limit disturbance. Receiving many notifications in a short interval or during inopportune times can overwhelm and irritate users [24-26], which may cause them to turn off notifications or even uninstall the app. Therefore, the notifications were designed such that they are never sent more than twice per day, never sent after 7 PM, and sent only when the user seems to have abandoned the Floodlight Open tests.

Statistical Analysis

The quantitative UX analysis was conducted on data available in the publicly available data set [15]. As information on self-declared MS status was collected only after the Floodlight Open app was activated, both participants with MS and participants without MS were included in the analysis on the effectiveness of the more seamless user journey. In contrast, the analysis on configurable reminders and push notifications was limited to participants with MS to keep the analysis as consistent as possible with the quantitative UX analysis.

To study the effectiveness of the more seamless user journey, the conversion rate, which is the proportion of participants who successfully completed the registration and activation process, was compared across 2 cohorts. The first cohort followed the original user journey without the deep-linking feature, whereas the second cohort followed the more seamless user journey that took advantage of the deep-linking feature. Because this UX design feature was released on a separate schedule for the iOS and Android versions of the Floodlight Open app, the first cohort included data collected from November 12, 2018, when the activation dates of the FLO IDs were first logged, through July 17, 2019, when the more seamless user journey along with the deep-linking feature was first implemented. The second cohort included data from all participants who registered from October 21, 2019, when the more seamless user journey was fully implemented in the United States, through November 2, 2021, when the study closed in the United States. The gap between July 17, 2019, and October 21, 2019, was required, as the more seamless user journey feature was released on different schedules for iOS and Android platforms.

The impact of introducing push notifications and configurable reminders on UX (ie, user engagement and retention) was assessed through continuous app usage time, inactivity time, test completion rate, and user retention rate. The continuous usage time is the number of consecutive days on which a participant used the Floodlight Open app, namely, on how many days in a row they performed the Floodlight Open tests.

The inactivity time, by comparison, is the time interval between 2 continuous use times during which a participant did not perform any test, that is, how many days in a row a participant waited before returning to perform a test. This metric is strongly affected by the frequency with which participants performed the tests and the likelihood of participants returning to the tests. The nature of the distribution can guide, for example, the choice of successful engagement methods that can further increase user engagement and retention. For both continuous usage time and inactivity time, the probability of n days of continuous usage and inactivity, respectively, was computed.

The test completion rate is defined as the fraction of participants performing at least n test series (DA, all daily activities included in the fixed sequence except the SBT and UTT [DA_x], or 2MWT) since the activation of the Floodlight Open app. The completion rate is a common measure for assessing the effectiveness of UX features, that is, for assessing whether the tasks performed by users achieve specified goals in terms of accuracy and completeness in a specified context of use [27]. It does not consider how the goals were achieved but only the extent to which they were achieved. The higher the test completion rate is, the more engaged the users are and the more likely they are to come back and, in this case, perform the assessments. The test completion rate is calculated over the entire duration of the assessed period and does not distinguish between intermittent and continuous usage.

The user retention rate, by comparison, is the fraction of participants who returned to perform the Floodlight Open tests n days after the activation of the Floodlight Open app. The retention rate is extensively used to measure the success of smartphone apps, with higher retention corresponding to higher adoption and level of engagement [28,29].

These UX metrics (continuous app usage time, inactivity time, test completion rate, and user retention rate) characterize the UX with the Floodlight Open app and provide insights into how committed the participants are. Because taking the gait and postural stability tests require both time and space (see the Qualitative UX Analysis section), 3 separate test series were considered for each of these metrics: DA (ie, all active tests administered in the fixed sequence: DMQ, IPS, IPS Digit–Digit, PT, DaS, SBT, and UTT), all DA except the gait and postural stability tests (DA_x; ie, DMQ, IPS, IPS Digit–Digit, PT, and DaS), and 2MWT.

For each test series, the complementary cumulative distributions of the 4 UX metrics were compared in MS participants using data collected before (November 12, 2018, through October 21, 2019) versus after (February 19, 2020, through November 2, 2021) the configurable reminders and push notifications were introduced. The gap between November 12, 2018, and October 21, 2019, was necessary due to the different release schedules of reminders and notifications for the iOS and Android versions. Hence, during this period, reminders and notifications were available to some, but not all, MS participants. It is possible that individual participants were included in both observation periods if they took the Floodlight Open tests during both periods. Because the different durations of these 2 periods might impact the continuous usage times and the inactivity times, both UX metrics were assessed over the first 343 days of data collected from each participant.

Results

Participants

Until November 2, 2021, when the study closed in the United States, the Floodlight Open app was downloaded 5225 times, including 4240 (81.15%) times on iOS devices and 985 (18.85%) times on Android devices, across the 17 participating countries. The baseline demographics of the US participants with MS included in the qualitative and quantitative UX analyses on reminders and notifications are presented in Table 3.

Qualitative UX Analysis

A total of 22 US participants were enrolled in the qualitative substudy, of which 15 (68%) were interviewed. Between 7 and 19 participants are considered adequate for qualitative research [30].

The qualitative UX analysis reveals key points for improving the UX with the Floodlight Open app (Table 4). While many participants did not face any difficulties downloading the Floodlight Open app and were willing to perform the tests daily, some were unaware or confused about the activation process despite instructions being provided on the study’s web portal (see Need for a More Seamless User Journey section in Table 4). Other participants reported that they did not complete the activation process due to distractions or the lack of reminders to complete this step. Here, a more seamless journey from registering on the web portal to activating the Floodlight Open app could improve the UX. In addition, participants who successfully activated the Floodlight Open app self-reported that they used it for 2 to 30 days and agreed that screen contents, prompts, and information were easy to see and understand. However, they also reported that they often forgot to perform the tests or felt it was not always feasible to perform all the tests in one setting. Therefore, configurable reminders and push notifications would have been beneficial (see bottom section in Table 4). Such UX design features could be used to set up reminders to perform the tests or to stop the current test and plan to complete the tests later. Furthermore, participants were not always available to perform tests that required them to stand or walk:

I can only proceed so far because I’m in the car waiting for my kid; I can’t stand and balance...or do the walking back and forth and the balancing stuff.

The part that is difficult to get done is the 2-minute walk. To find the time...is sometimes hard for someone like me who is more active.

Certain tasks...were a little too long. How many more lines do I need to draw? How many more tomatoes can I pinch?

Consequently, in the subsequent section on quantitative UX analysis, UX with the Floodlight Open app was investigated with respect to DA, DA_x, and 2MWT.

Quantitative UX Analysis

User Journey

Of the 1089 US participants, 608 (55.8%) with or without MS (self-declared MS disease status was not available for this analysis, as this information was collected only after the successful activation of the app) followed the original user journey without the deep-linking feature, and 481 (44.2%) participants with or without MS followed the newer, more seamless user journey with the deep-linking feature. The flow diagrams in Figure 4 show the conversion rates from creating an account to activating the app for both cohorts. In both cohorts, all participants who created an account completed the registration process, provided informed consent, and enrolled in the study. However, Figure 4 shows that introducing the new user journey improved the conversion rate of the participants who succeeded in activating the app (359/481, 74.6% with the more seamless user journey vs 328/608, 53.9% with the original user journey).

‎

Configurable Reminders and Push Notifications

The analysis on configurable reminders and push notifications included 172 participants with MS to whom this UX design feature was available and 350 participants with MS to whom this feature was not available (Table 3). The cumulative distributions of the continuous usage times of participants with MS using the Floodlight Open app with reminders and notifications and those using the Floodlight Open app without reminders and notifications are plotted in a log-log scale for DA, DA_x, and 2MWT in Figure 5A. The introduction of reminders and notifications increased the continuous app usage times. The longest continuous usage times with reminders and notifications versus those without reminders and notifications were 103 versus 13 days for DA, 224 versus 61 days for DA_x, and 65 versus 12 days for 2MWT. Similarly, reminders and notifications increased the proportion of participants using the app for at least 3 consecutive days (DA and DA_x: ~30% vs ~12%; 2MWT: ~30% vs ~20%).

‎

Figure 5B presents the inactivity time distributions from participants using the Floodlight Open app with configurable reminders and push notifications and those using the Floodlight Open app without configurable reminders and push notifications on a log-log scale for DA, DA_x, and 2MWT. These distributions were similar for participants with reminders and notifications and those without reminders and notifications for DA, DA_x, and 2MWT, exhibiting an approximate power law with a coefficient of 0.96 in both cohorts. The longest inactivity times with reminders and notifications versus without reminders and notifications were 145 versus 263 days for DA, 138 versus 309 days for DA_x, and 181 versus 263 days for 2MWT.

Figure 5C presents the test completion rate distributions for DA, DA_x, and 2MWT. These distributions show that the introduction of reminders and notifications resulted in higher test completion rates, except during the initial onboarding phase when DA and DA_x completion rates were similar in both cohorts. During this onboarding phase, day-1 test completion rates with and without reminders and notifications were 66% (113/172) and 66.9% (234/350), respectively, for DA and 81.3% (140/172) and 86% (301/350), respectively, for DA_x. These test completion rates are expected to be <100% because not all participants who activated the Floodlight Open app proceeded to perform at least 1 DA or DA_x series. However, the effectiveness of introducing reminders and notifications became evident after this initial onboarding phase (ie, after the first DA series and after the third DA_x series; Figure 5C). When considering the 2MWT instead, participants with reminders and notifications showed higher test completion rates than participants without reminders and notifications (41.5% [72/172] with reminders and notifications vs 34.8% [122/350] without reminders and notifications) on the first day itself. Across all 3 test series, the largest number of test series completed increased with the introduction of reminders and notifications (279 vs 64 for DA, 283 vs 126 for DA_x, and 302 vs 76 for 2MWT).

Figure 5D presents the user retention rate distributions. During the initial onboarding phase, user retention rates were comparable between participants with reminders and notifications and those without reminders and notifications. On day 1, DA user retention rates were 65.7% (113/172) and 61.7% (216/350), respectively, and DA_x user retention rates were 79.6% (137/172) and 82.8% (290/350), respectively (Table 4). DA_x user retention rates continued to be comparable between both cohorts for approximately the first 10 days. After this onboarding phase, however, the effectiveness of reminders and notifications in improving user retention rate became evident. Interestingly, after the high drop-off observed during the first few days after the activation of the app (3 days for DA and DA_x and 2 days for 2MWT), all user retention rate distributions obtained with reminders and notifications plateaued up to approximately day 30. On day 30, user retention rates were considerably higher in participants with versus participants without reminders and notifications (53/172, 30.8% vs 34/350, 9.7% for DA; 53/172, 30.8% vs 60/350, 17.1% for DA_x; and 39/172, 22.6% vs 22/350, 6.2% for 2MWT; Table 5).

Discussion

Principal Findings

Smartphones enable out-of-clinic assessments of chronic neurological diseases. Despite the rapidly increasing number of mobile health care apps available for consumers’ self-care, there is a paucity of research into the UX of DHTTs for MS. In this paper, we present our qualitative and quantitative UX analyses of the Floodlight Open app and demonstrate that the adoption of key UX design features markedly improved user acquisition, engagement, and retention. Understanding how the presence or absence of specific UX design features affects participants’ experiences offers important guidance for the refinement of an existing or the design of a new DHTT. Therefore, UX design features are a significant consideration when designing and evaluating such tools.

Assessing MS symptoms over time with the Floodlight Open app requires specific user stimulation and guidance through seamless and flexible user journeys enhanced by engaging features to improve the overall UX. By considering participants’ feedback collected in the quantitative UX substudy, we enhanced the onboarding process by implementing a more seamless user registration and app activation journey and improved the UX through configurable reminders and push notifications. We derived several distributions from real-world data that describe key aspects of participants’ behavior in their environment. Such measurements and their statistical analysis might be used to help describe realistic user behavior and design health care apps similar to our Floodlight Open app. Our findings allow us to gain a better understanding of users’ experience in such settings and to make informed design decisions for self-monitoring mobile apps that support people with MS.

Users’ first experience with the registration to the Floodlight Open study and the activation of the study app is important to increase user acquisition. Furthermore, the first few days after activating the app are similarly critical for long-term engagement and retention. For instance, user acquisition, engagement, and retention can all be improved by guiding users throughout the Floodlight Open study registration, app activation process, and daily activities, reminding them about the daily activities to be performed and allowing them to plan when to perform the daily activities. Our results demonstrate that a seamless UX is essential to minimize the rate at which users drop out along the end-to-end journey. With a seamless user journey, users are led along the registration-activation journey and are not distracted by unrelated content or activities, resulting in a higher conversion rate from creating an account to activating the Floodlight Open app.

Configurable reminders and push notifications can help keep the participants engaged with mobile health care apps, such as the Floodlight Open app, and help retain them beyond the initial onboarding phase. Our findings show that continuous use times, test completion rates, and user retention rates are generally higher with configurable reminders and push notifications than without configurable reminders and push notifications, whereas inactivity time is comparable between the 2 cohorts. This improvement in user engagement is most evident for 2MWT from the first day and for DA and DA_x after the initial onboarding phase, during which the participants are still exploring the Floodlight Open app. Therefore, to fully optimize the UX, the app should ask the users to accept and receive reminders and notifications only after this initial onboarding phase [32]. Interrupting users with notifications too soon after the activation of the app may create frustration and may ultimately lead them to abandon the app. The app should, therefore, minimize the mental and physical interaction efforts during the first few days of use.

Comparing the different test series, we noted slightly worse user retention for 2MWT than for either DA or DA_x. Taking the 2MWT requires the greatest effort, which is in line with some dissatisfaction reported in a previous proof-of-concept study [33]. It is possible that the study participants did want to take the 2MWT but did not find a suitable opportunity to take it each day. In fact, participants in Floodlight Open were found to persistently take the 2MWT at least once per week [15], which suggests that a more flexible or less frequent assessment schedule could be beneficial.

Comparison With Prior Work

Although several other smartphone apps are available for people with MS (for a review on MS apps, refer to the study by Howard et al [34]), to our knowledge, this is the first analysis to assess the impact of UX design features on user engagement and retention. Nonetheless, previous analyses have shown that user engagement and retention remain two of the barriers to the long-term successful deployment of DHTTs (for reviews, refer to works of Pratap et al [6] and Amagai et al [35]). Several UX design features have been suggested to improve the overall UX, including a more seamless user journey, configurable reminders, and push notifications [23,35-38]. However, differences in the definitions used to assess user retention (eg, user retention definitions based on the user simply opening or interacting with the DHHT [31] vs completing a test or series of tests [6]), limited statistical power due to small sample sizes [37], and differences in the observed study period [37] may make direct comparisons of user retention across different DHHTs challenging.

To compare like with like, we computed the user retention with the Floodlight Open app for the same observation period as that reported in the study by Rosenfelder [31] for medical as well as health and lifestyle apps. Both data sets indicate that the largest reduction of users occurs from days 1 to 3, whereas the reduction in user retention is low between days 7 and 30. This suggests that a significant proportion of users who will not use the Floodlight Open app (or other health care apps) in the long term will stop using the app within the first 3 days and that this dropout is likely to be observed within the first 7 days. These results also suggest that participants who use the app for >3 days are likely to use it for at least 30 days. A similar dropout during the first 7 days has also been previously reported for the whole study cohort of Floodlight Open [15].

Of note, our user retention rates are approximately 2- to 6-fold higher than those reported by Rosenfelder [31] (Table 5) despite using a more conservative definition of user retention (completing a test series vs simply interacting with or opening the app). It is conceivable that not all apps included in the report by Rosenfelder [31] feature the same UX design features as the Floodlight Open app does. However, favorable user retention rates were observed with the Floodlight Open app even when the app did not have configurable reminders or push notifications. This suggests that differences in the perceived benefit of the app might have also contributed to the favorable retention profile, although other reasons cannot be excluded [39]. Despite our favorable user retention and user engagement findings, a previous analysis showed that improvement in smartphone sensor data collection could be achieved through passive data collection methods [15].

Limitations

A few limitations are noted. First, while participants could turn off the configurable reminders and push notifications, this action was not logged. Instead, we compared participants who enrolled before with those who enrolled after these features were implemented. For the former cohort, we included only the data collected up to the implementation of reminders and notifications in our analyses to be able to assess the impact of reminders and notifications. Second, our results were influenced by the duration of the study and by incoming participants who activated the Floodlight Open app on different dates over the duration of the study and might not be part of the study for the same number of days. Besides, for events approaching the duration of the study, there is an artificially lower likelihood of observation, and events lasting longer than the study cannot be observed. Finally, the use of the Floodlight Open app in a clinical study other than the Floodlight Open observational study may boost user engagement and retention due to additional motivational incentives. Such clinical referrals have been shown to improve user retention [6]. However, information on which participants participated in such studies is not available, and without it, accounting for it is not possible.

Conclusions

We presented the qualitative and quantitative UX analyses of the Floodlight Open app, a DHTT for MS. Learnings from the qualitative UX analysis led to the implementation of a more seamless user journey and the introduction of configurable reminders and push notifications. These US design features improved user acquisition, user engagement, and user retention. The continued assessment of UX and improvement of UX design features are critical steps in optimizing the long-term adoption of the Floodlight Open technology and similar DHTTs.