Data were collected during a clinical study approved by the French Institutional Review Board (approval number: 2018-A03129-46) and conducted by Withings from 2020 to 2022. The inclusion process for the study was conducted prospectively by a third party, a general-purpose survey company owning listings of self-declared individuals from the general population. Participants were contacted based on the inclusion and exclusion criteria. In particular, the contacted persons had to have no preexisting medical conditions that would necessitate a PSG examination, and inclusion criteria aimed to ensure a balanced distribution of BMI within each age group. The cohort consisted of a consecutive series of 194 volunteers who provided informed consent. To guarantee data privacy, a unique, randomized anonymization code was assigned to each participant (used for both WSA and PSG data correspondence). Withings only had access to these pseudonymized datasets and never had access to direct identifiers. No compensation from Withings was provided to the participants.

WSA is designed to identify sleep stages over 6-minute windows. As a result, this analysis focuses on evaluating WSA’s performance in detecting stages longer than 6 minutes. It should be kept in mind that sleep stages shorter than this are ignored by the device in this study. In addition, WSA does not distinguish stages N1 and N2, so for the aim of comparison, the consensus sleep stages N1 and N2 obtained from the PSG were combined into the “light” category. Rapid eye movement (REM) and awake stages of the WSA and the consensus are directly comparable, and N3 is compared with the stage “deep sleep” from the WSA. The resulting stages are illustrated in Figure 2.

Two independent certified sleep technologists annotated the PSG signals in 30-second epochs, classifying sleep stages as N1, N2, N3, REM, and awake, following the American Academy of Sleep Medicine (AASM) scoring manual [48]. A third professional reviewed the 2 initial readings and established a consensus for reference data. Sleep stage comparisons were performed over windows of 30 seconds. To evaluate WSA’s performance in detecting sleep stages longer than 6 minutes, consensus data were smoothed by selecting for each 6-minute window the stage with the highest occurrence of 30-second epochs. For more details about the procedure, please refer to Section A of Multimedia Appendix 1.

In addition to standard sleep quality measures calculated over the night, such as TST and the duration of each stage, evaluating stage classification required synchronizing epoch-by-epoch data from both devices over time. Synchronization of the clocks of the devices was obtained by maximizing the correlation of the breathing signals of the two devices [51]. To verify the robustness of our breathing-based synchronization, we assessed the impact of temporal shifts (up to ±90 seconds) on classification performance [39,52]. This sensitivity analysis revealed negligible variations, with the SD for sensitivity and specificity across all sleep stages remaining below 0.005. Details of the procedure can be found in Section A of Multimedia Appendix 1.

An example hypnogram is presented in Figure 2, showing the smoothed, PSG-based consensus in red and WSA sleep stages in blue. WSA indicates bed absence/presence, marked respectively by a red and yellow background, information which is absent from the PSG data. The consensus annotations use instead the events lights-on/off detected by the ambient light sensor of the PSG as proxies for bed-in and bed-out events. For further information on challenges arising from using lights-off/on timestamps as proxies for bed presence, please refer to Section A of Multimedia Appendix 1.

Standard sleep quality measures used for this analysis and based on the AASM scoring manual [48] are the following: total sleep time, time spent in each sleep stage (TIREM, TILight, TIDeep, and TIWake), proportion of the time spent in each sleep stage compared to total sleep time (PIREM, PILight, PIDeep, and PIWake), and number of episodes in each sleep stage (NEREM, NELight, NEDeep, and NEWake) [50]. Each of these measures has been computed by each device (PSG and WSA) and then compared using the MAE and a Bland-Altman analysis, described in Section B of Multimedia Appendix 1.

Epoch-by-epoch agreement is assessed using standard classification metrics. Subgroup analyses were conducted to determine whether WSA performance is affected by demographic, clinical, and environmental factors. Details regarding the evaluation metrics and subgroup analyses are available in Sections B and C of Multimedia Appendix 1.

To compare the median MAE between different groups (categories of a given metadata as listed in Table S3 in Multimedia Appendix 1), a Kruskal-Wallis test has been performed on the MAE obtained on all the metadata described in Table S3 in Multimedia Appendix 1 for each of the following sleep quality measures (TST, TIREM, TILight, TIDeep, PIREM, PILight, and PIDeep) and for the number of episodes in each sleep stage.

To compare the MAE variance between different groups, a Brown-Forsythe test has been performed on the same data for the same sleep quality measures. The Holm-Bonferroni correction is used after both statistical tests to address the issue of multiple comparisons.

Demographic and clinical data are summarized in Table 1 for the population included in the analyses.

A total of 117, mostly healthy, participants were included in the analysis. The cohort comprised 59% (n=69) women and 41% (n=48) men, with a mean age of 39.9 (SD 11.4) years, ranging from 19 to 69 years. The average BMI was 23.9 (SD 3.3) kg/m².

The average sleep quality in this population is considered mildly impaired, with a mean PSQI score of 6 out of 21. The SD of 2.9 highlights significant variability.

The most common mattress types were foam (n=41, 35%), spring (n=29, 25%), and latex (n=24, 21%). Around 55% (n=64) of participants reported having a bed partner. Sensor disturbances were reported by 46 (39%) participants due to the PSG device, while no disturbances were reported regarding the WSA nearable.

Figure 3 presents the confusion matrix containing the epochs across all nights, and Table 2 gathers sleep stages classification results on the whole dataset.

Figure 4 shows the TST estimated by the WSA versus the PSG. The slope of the linear regression is 0.98 (P<.001) and the R² value is 0.89, with 38% of outliers, indicating a moderate linear relationship with PSG.

WSA deep sleep was not correlated to time spent in deep sleep (slope estimate of 0.05, P=.7, Figure S4 in Multimedia Appendix 1). The WSA estimates of light sleep and REM were moderately correlated with PSG values, with respective slopes of the regression lines of 0.4 (P<.001, Figure S5 in Multimedia Appendix 1) and 0.65 (P<.001, Figure S6 in Multimedia Appendix 1).

Precision of the WSA estimates is measured with the MAE. The MAE values of total sleep time (equal to the MAE of wake duration because the sum of TST and wake duration is the recording time), REM duration, light sleep duration, and deep sleep duration are 44 min, 35 min, 1 h 33 min, and 1 h 4 min, respectively, with average PSG values of 7 h 15 min (3 h 03 min for awake duration), 1 h 43 min, 2 h 57 min, and 2 h 35 min, respectively. Corresponding 95% CIs are reported in Table 3. The average number of episodes in each sleep stage and the average number of stage changes can be found in Table 4. The WSA tends to underestimate the number of episodes for each sleep stage and is less sensitive to transitions between epochs.

Additional scatter plots and regression analyses of these metrics can be found in Section D of Multimedia Appendix 1.

The Bland-Altman plots are presented in Figures 5 and 6. Biases are summarized in Table 3.

In Figure 5, the Bland-Altman plot for TST does not reveal any proportional bias with the duration of the night. The mean error is 20 minutes. The wide limits of agreement show that there is a substantial amount of variability between PSG and WSA. By contrast, the average error in REM, light, and deep sleep durations depends on the duration of the phase, therefore exhibiting proportional bias. Light sleep is on average overestimated for any duration, deep sleep is on average underestimated for any duration, and REM is on average underestimated for short durations and overestimated for long durations.

The mean error (bias) in light sleep duration is important, with 1 h 21 min. Biases in REM and deep sleep are −15 min and −46 min, respectively. The mean error in TST is therefore 20 min.

No significant differences were found in the median or variance of the error across most groups for all the quantitative sleep variables of Table 3. This includes comparisons based on gender, BMI categories, age groups, sleeping arrangements, PSQI scores, self-reported sleep quality, mattress thickness, and night awakenings. However, significant differences in median total sleep time and wakefulness proportion were observed within the PSQI sleep duration component subgroup. Median MAE of TST is respectively 21 min and 1 h 11 min for good and bad sleep duration subgroups, and median MAE of wakefulness proportion is respectively 3% and 12% for good and bad sleep duration subgroups (N=117). This finding suggests that distinguishing sleep from wake is a harder task for the WSA when the participant has subjective short sleep duration (<6 h), highlighting an area for further investigation. For a comprehensive overview of the statistical tests and detailed results, please refer to Multimedia Appendix 2.

As an illustration, Figures 7-9 show the boxplots of TST MAE of WSA compared to PSG, stratified by age groups, BMI ranges, and whether the participant was alone in bed. They suggest that the WSA performance did not depend on these factors. Boxplots for TST MAE for the other subgroups can be found in Section D of Multimedia Appendix 1.

Each 30-second epoch of the PSG data was independently annotated by two reviewers. The mean κ value for sleep stage determination between reviewers before reaching consensus is 0.71 (SD 0.11), closely aligning with literature findings (κ=0.76 in [53]), while it is 0.81 (SD 0.17) for sleep-wake classification.

In the analysis of reviewer agreement on sleep stage classifications, the primary disagreements occur between light and deep stages, as well as between light sleep and awake/REM stages (Figure 10). This indicates that even expert reviewers, considered the gold standard, face challenges in accurately distinguishing these stages due to their inherent similarities and subtle physiological differences. This inherent difficulty in distinguishing light, deep, awake, and REM sleep stages by human experts is reflected in WSA, which was trained on PSG data labeled using a similar protocol. It is likely that the imprecision of the gold standard is passed on to the device.

Since this analysis does not involve the WSA or the PSG data, but only the participants’ answers to the questionnaires, it was conducted on the 186 participants for whom reported sleep quality data were available. Results are given in Table 6.

Despite the removal of the nasal cannula and the blood oximeter from our home PSG test, the sleep test was associated with significant discomfort in 41% (77/186) of the participants during the night of the PSG home test, compared to 0% (0/186) in the subsequent night with the WSA only. Fifty-two percent (97/186) (resp. 20%, 38/186) of the participants reported a degraded sleep quality on the night of the PSG (resp. without the PSG).

Figure 12 illustrates the changes in sleep quality among participants over 2 consecutive nights. Participants generally experienced better sleep without the PSG sensors. Notable improvements were observed in 52% (97/186) of cases, with transitions from fairly good to good (26/186, 14%), and from poor to fairly good (31/186, 17%), good (12/186, 6%), or very good (10/186, 5%). Additionally, some participants reported significant improvements from very poor to very good sleep, and others from poor to fairly good. These improvements may be influenced by several factors, including the removal of PSG sensors, which can be intrusive and uncomfortable, potentially disrupting sleep. Furthermore, the awareness of being recorded might alter participants’ sleep behavior, either positively or negatively. The use of unfamiliar devices could also play a role, as participants may initially experience discomfort or anxiety, which diminishes over time. External conditions such as noise and room temperature, which were not controlled in this study, could also significantly impact sleep quality. Therefore, while the data suggest improvements, it is crucial to consider these potential confounding factors when interpreting the results.

Table 6 highlights that participants slept as well with WSA as they typically do. On the second night, 59.7% (111/186) of participants reported the same sleep quality, 16.7% (31/186) reported better sleep, and 3.2% (6/186) reported much better sleep compared to their usual experience. This suggests that Withings sleep monitoring devices do not negatively impact sleep quality and may even enhance it for some users.

The evaluation of the WSA for sleep stage detection in a cohort of 117 participants in a home setting revealed several key insights. The device demonstrated a high sensitivity of 93% for detecting sleep, underscoring its strong capability to accurately identify sleep periods. However, the specificity was moderate at 73%. The duration of the recording before actual sleep onset and after wake-up significantly affects observed performance of wake detection, with shorter periods inflating specificity, accuracy, and Cohen κ. This occurs due to a shortening of quiet wake periods hard to distinguish from sleep (reading or screen time in bed, for instance), leading to fewer false positives while true negatives remain relatively constant. Conversely, overly long (eg, 24-hour) recordings, dominated by easily identifiable diurnal wakefulness, also tend to inflate specificity. In both cases, the performance of the evaluated device is overestimated. The most clinically relevant and challenging scenario for performance evaluation involves analyzing data encompassing the entire in-bed period, including calm wakefulness before sleep onset and after awakening. While AASM guidelines suggest using lights-off and lights-on as proxies for night boundaries, this approach truncates crucial pre– and post–sleep wake data, therefore introducing a bias in TST and classification performance endpoints. To mitigate this, we used all available PSG annotations, irrespective of lights-off and lights-on times, to provide a more comprehensive and unbiased assessment of performance across the full in-bed duration.

The classification of sleep stages (into wake, REM, light, and deep phases) is more challenging. The confusion matrix (Figure 3) shows that 40% of deep sleep phases, 37% of REM sleep, and 20% of wake phases were misclassified as light sleep by the WSA. As a result, the corresponding sensitivities were rather low (0.56, 0.53, and 0.73, respectively). Thirty-two percent of light sleep stages were misclassified, in similar proportions, into REM, deep sleep, and wake. In consequence, (1) the REM and deep sleep total durations were underestimated (by approximately 30% and 15%), while light sleep duration was overestimated by almost 48% (biases in Table 3), and (2) the specificities for wake, REM, and deep sleep are very high (0.93, 0.94, and 0.94, respectively), while the specificity for light sleep is moderate (0.69).

We showed that these misclassifications of WSA mirror the disagreements between PSG reviewers, who mainly disagree on light versus all the other sleep stages (Figure 11). WSA classification was indeed significantly better on epochs where reviewers agreed (79% of the epochs) than on epochs where they disagreed. WSA’s sleep stage classification uses an algorithm trained by supervised learning on PSG data annotated by technician experts, so it is expected that WSA performance is limited by the interreader variability.

Consistent with existing validation studies [33,44], we also report accuracy and Cohen κ. We caution that these performance metrics are strongly dependent on the prevalence of the classes (here, wake, sleep, REM, deep, and light), which creates difficulties in comparing results between studies. In particular, for sleep studies, two factors modify the prevalence of the classes significantly. First, there is no standardized definition of the beginning of the night. Second, the choice of the reference device has some importance, causing more or less sleep disturbances.

With these caveats in mind, the WSA achieved an overall accuracy of 87% for sleep-wake distinction, which is promising for a contactless home monitoring device. Despite this, the accuracy for classifying specific sleep stages was moderate, with REM sleep at 53%, light sleep at 68%, and deep sleep at 56%. The overall accuracy for sleep stage classification was 63%, with a Cohen κ of 0.49.

Notably, the WSA’s performance was consistent across various demographic subgroups, including different age and BMI categories. Importantly for a device under the mattress, there was no difference between individuals who slept alone and those with a bed partner, nor between different types or thicknesses of mattresses. Other groups analyzed included sex, PSQI scores, self-reported sleep quality, and whether individuals woke up during the night, all of which also showed no significant differences in MAE, except for wake proportion by subjective sleep duration group. This consistency suggests the device may be useful for a wide range of participants.

The rapid adoption of new sleep technologies in consumer and clinical settings demands rigorous performance evaluation [64]. To improve the reliability and comparability of sleep research, we describe our methodology in detail and with transparency, an approach that is supported by recent guidelines advocating for uniform analysis pipelines [41,42,44]. Our study was designed to resolve methodological issues frequently found in the literature.

The study’s strength lies first in its diverse sample of over 100 participants, who are for the most part healthy and with diverse demographic characteristics. Additionally, the inclusion of relevant factors that may negatively impact the device’s performance, like the presence of a bed partner and variations in mattress type and thickness, further enriches the dataset, allowing for a comprehensive evaluation of the device’s robustness across different real-life conditions in a healthy population. This allowed us to conduct meaningful subgroup analyses.

A second strength of the study is the method of PSG analysis. Unlike [44,55-58], which opted for a single reviewer, or two reviewers without consensus as in [37], our study is aligned with [33,38] by implementing a double review process with adjudication by a third reviewer in case of disagreement. This consensus methodology is preferable since it enhances the reliability of the sleep stage classification. In particular, we have shown that reviewers often err by misclassifying light sleep stages, a finding consistent with previous research reporting that the interrater reliabilities for light and deep stages were moderate [61]. A reliable assessment of devices’ performance in sleep stage classification therefore requires a strong methodology to establish the ground truth.

Third, with a view to limit the participants’ discomfort, we chose to perform PSG at home with a limited number of sensors, without the nasal cannula and oximeter. This setup reduces the sleep disturbances typically associated with lab-based PSG, allowing for a more natural assessment of sleep patterns. This limited set of sensors is sufficient for the evaluation of TST and sleep stage classification, with the trade-off that sleep onset, time in bed, and wakefulness after sleep onset cannot be assessed. Yet, significant sleep disturbances were reported by the participant during the PSG test, underscoring the challenges of completely undisturbed monitoring even in home environments. Despite this, the study offers a realistic evaluation of the WSA’s capabilities under typical sleeping conditions.

Fourth, we aimed to give a comprehensive report of the device’s performance. Collecting epoch-by-epoch data enables a granular examination of sleep patterns, the consistency of measurement errors over time, and thus enables an accurate assessment of the reliability and utility of sleep monitoring devices in real-world settings. We emphasized the need for such comprehensive reporting and the elements of the protocol that may bias the performance estimates. Thus, transparency requires a presentation of the results including all the performance endpoints, each presenting a different facet of the device’s performance [41-43].

In conclusion, given these methodological strengths, our study provides valuable insights into the WSA sleep stage algorithm’s performance in real-world settings.

Despite our efforts for a solid methodology, the study faced several challenges.

In this study, we aimed to (1) assess the performance of a contactless device for sleep stage identification against the gold standard PSG, (2) benchmark it against other comparable devices, and (3) better understand the limitations of such devices. Our paper contributes to those, while also laying out important methodological issues.

This study contributes to the growing body of evidence supporting the utility of nearables for longitudinal sleep monitoring. The WSA’s contactless, undermattress design facilitates unobtrusive data collection in free-living conditions and demonstrates sleep-wake detection performance comparable to established consumer devices. While precise sleep stage classification remains a technical challenge, our results highlight the importance of testing on large, diverse, challenging datasets to capture real-world variability. This research underscores the need for standardized performance assessment methodologies to enable reliable comparisons across devices and studies, offering insights for future device design and guidance for utilization, advancing the goal of improved sleep monitoring for the general population [71].