Assessing an individual’s mobility can provide meaningful insights into their general health status. In clinical settings, mobility is a fundamental factor to define prognosis and care as it is closely associated with a wide array of health outcomes [1-3]. However, accurate and reliable quantification of mobility in real-world settings remains challenging because self-reported data from instruments such as the International Physical Activity Questionnaire can be biased by limited recall and social desirability [4,5].

The interest in quantifying physical activity using wearable devices has recently increased, as these technologies can collect objective individualized data [6]. Wearable sensors have been incorporated into clinical studies across different disease states to enable movement analyses and the quantification of discrete physical activities to develop clinically meaningful end points [7,8].

Yet, to cement their research utility, two aspects of these digital measurements need to be properly characterized: (1) the accuracy with which a digital measurement reads the parameters of interest [9] and (2) the amount of aggregated data needed to reliably capture an individual’s underlying behavioral state, minimizing noise related to natural variability, which usually translates into an aggregation time period for data collection (time to reliability). Although accuracy is always a critical aspect in the characterization of a measure’s performance, time to reliability tends to be ignored, even though it is key for establishing fundamental study design specifications (eg, collection time periods and length of wear time per day), defining baselines, or computing power calculations for the detection of intervention effects or other changes.

Studies characterizing the performance of digital walking measures often focus on step count. However, the literature around these studies shows considerable heterogeneity across designs and some notable limitations. First, analyses tend to rely on truth labels originated by participants’ self-reports, short-term close monitoring [10-12], or from reference devices with suboptimal accuracy (mean absolute percentage error >20%) and with the same body placement as the investigational devices, which would bias agreement results [13]. Second, these studies are often conducted in artificial laboratory environments, which inherently limit behavior range and are susceptible to subjectivity, assessment bias, and unreliability [14-16]. Third, reliability characterization for investigational digital measurements is often absent from studies, despite having been acknowledged as an important element for the validation of clinically important research metrics, such as patient-reported outcomes [17]. Beyond step counts, there have been studies that have used other digital measures (eg, walking intensity captured by the peak 30-minute cadence) to generate clinical insights but without full characterization of their performance [18-29].

In a previous study, we developed an algorithm that accurately classifies ambulatory status from data collected from a wrist-worn device, characterizing its performance across diverse demographic groups in a real-world setting [30]. Further, results from a substudy of an interventional randomized phase 2 trial demonstrated that digital measures of physical activity (step count and ambulatory time) could be sensitive to treatment effects in patients with Lewy body dementia [31].

Herein, we report on the development of a series of measures that can capture walking behavior comprehensively, characterizing their analytical performance in accuracy and reliability. These measures included (1) step count, (2) walking bout duration, (3) number of total walking bouts, (4) number of long walking bouts, (5) number of short walking bouts, (6) peak 30-minute walking cadence, and (7) peak 30-minute walking pace. To characterize their accuracy, we compared the measure readouts generated from a study device with highly accurate truth labels from an ankle-worn reference device in healthy volunteers. To characterize their reliability, we developed a novel approach to calculate the aggregated time required to reach a reliable readout (time to reliability) for each measure.

This report expands upon prior research [30,31], presenting a comprehensive application of an algorithm that captures step count and other aspects of mobility, such as walking cadence and bouts. We characterized the accuracy and reliability of this comprehensive set of digital walking measures from users wearing a wrist-worn device in real-world environments. We showed that these measures of walking reached reliable readings at around 16 days of wear time, and their levels of agreement with the reference device, measured by ICC, ranged between 0.7 and 0.9, a performance that supports their deployment in clinical trial settings with confidence.

Mobility and walking behaviors represent meaningful aspects of health, known to be associated with quality of life in general and clinical prognosis in specific settings [33-35]. Therefore, improved methods to measure mobility and walking behaviors have the potential to improve clinical care and clinical trial efficiency. One of the goals of our research is to build accurate tools to objectively quantify the aspects of walking behavior and extract clinically meaningful information in discrete populations of interest. In prior work, we have developed algorithms whose outputs (measures for step counts and ambulatory time) demonstrated sensitivity to treatment effects in patients with Lewy body dementia [31]. We have also characterized the accuracy of an iteration of that algorithm in a cohort of diverse individuals in the real world [30].

To our knowledge, this report is the first to characterize the amount of data required to ensure that a digital measure is reliable in a real-world setting. We developed a novel analytic approach to characterize time to reliability, that is, the time needed for a measure to reach a degree of stability. Time to reliability is an important consideration to inform the design of clinical studies tracking real-world data, as it relates to specific metrics of interest. In this study, the time to reliability overall for all measures was ≤16 days (ICC ≥0.75 between nonadjacent readouts for all measures at day 16, Figure 4). This study included healthy individuals; in a clinical context, we anticipate that the stability of any given measure over time will be dependent on the type and severity of the disease of interest. It is reasonable to speculate that a mostly healthy cohort may demonstrate more variability and a larger distribution of walking behaviors than a cohort with disease burden, and this necessitates further research.

Most importantly, the algorithms developed to quantify daily step count and measures related to walking cadence and bouts were found to be accurate (agreement between the readouts from the study device and a highly accurate reference device ranged between ICCs of 0.7 and 0.9 for all measures, Table 2). Our study approach captured the measures of walking behavior in a real-world setting, over multiple days, to closely resemble actual use cases.

Considering the exponential growth of research on wearable sensors and related devices in recent years, it is important to place the capabilities described in this report in that context. In this work, we incorporated several key innovative approaches to address shortcomings present in comparable studies evaluating interdevice agreement. Prior studies have used colocated investigational and reference sensors (eg, 2 wrist-worn devices). But because of the known potential errors associated with body placement when capturing walking-related data [36-39], colocation could be vulnerable to bias toward overestimating performance. Our approach sought to mitigate that by using a highly accurate but pragmatic and ankle-worn source for ground truth labels. Further, most studies have narrowly focused on step counts [10-16] for short time periods in controlled laboratory environments (eg, only a single day in real-world settings), or when investigating walking bout and cadence or pace measures, they had a limited scope, with small samples of less than 40 participants [40-42] and short tests (sessions lasting 1 hour or less) performed in clinic. Our study addresses these existing evidence gaps, presenting a set of digital walking measures that are comprehensive beyond step counts and characterizing their analytic performance (accuracy and reliability) extensively, with data accrued throughout multiday periods and in the course of daily living activities. Moreover, given the research heterogeneity (comparisons of different devices, different ground truth sources, and with different analytic approaches), any direct comparison of study results side by side has to be done with caution, which highlights the need for standardization noted in professional statements in this field [6,9,43,44].

This study had limitations in regard to the participant population and the performance quality thresholds. First, our cohort was limited in size and consisted of generally healthy participants. Future studies may be needed to characterize the generalizability of the performance of these measures in populations with particular kinetic hallmarks (eg, neurological conditions, stroke, and trauma) or with mobility capacity issues (eg, cardiovascular or respiratory conditions). Our approach to determine time to reliability can be applied across studies in any therapeutic area and can guide study design requirements for wear time compliance. One aspect that will require attention is the optimization of actual compliance with hypothetical protocol specifications about wear time because this is a device intended for daily life use (for instance, our reliability analysis filtered participant-day data based on a threshold of 12 hours of daily wear time for 50% of the days over an evaluation period, but we did so retrospectively). Second, although we report on performance parameters, the definition of an acceptable performance (accuracy and reliability) quality threshold remains undefined in the field. We did not prespecify performance categories in this study, but, for instance, prior accuracy studies have categorized agreement ICC values 0.7-0.9 as moderate to good [40,45], and reliability studies for patient-reported outcomes have considered test-retest ICC >0.5 as acceptable [46]. Importantly, what constitutes a clinically meaningful change for each of the measures of walking will likely depend on the therapeutic area under consideration. Further research is also needed to address which of these measures can provide clinically relevant insights in a given population. This set of digital walking measures has the potential to convey comprehensive information beyond flat quantification (via step counts), about aspects such as maximal walking capacity, endurance, or activity patterns during daily living, which may have different relevance or sensitivity to detect status changes depending on the health setting (for instance, cardiopulmonary conditions, oncology, or neurology). Furthermore, the optimization of the clinical utility of these measures may require their aggregation into composite metrics. The potential complexity of this future research brings to the forefront the importance of establishing first a thorough understanding of their individual analytical performance, which this study does. We believe that the accuracy and reliability results detailed here are the first step to support the use of digital measures of walking as feasible and reliable end points in clinical studies.

In conclusion, we have developed algorithms that accurately quantify daily step counts and measures of walking cadence and bouts from users wearing a wrist-worn device in a real-world setting. Further, we have also developed a novel method for characterizing the time required for a digital measure to stabilize (time to reliability). Given the growing use of wearable sensors to measure aspects of health, these findings may guide practical implementation of these digital measures of walking behavior into clinical studies.