IntroductionBackgroundAlarm fatigue is a phenomenon where health care workers in intensive care units (ICUs) become desensitized to alarms of medical devices [1]. It can make ICU staff feel stressed, and it is a substantial risk to patient safety, as it can lead to alarms being missed or acknowledged with delay [2]. When implementing interventions or IT solutions [3] that try to remedy alarm fatigue, clinicians and clinical alarm researchers need a reliable way to assess whether they were successful. However, they have not yet agreed on a standardized way of measuring alarm fatigue [4,5], even though it was recognized more than 2 decades ago [6].
Solely analyzing an ICU’s alarm log data cannot serve as a measure of staff’s alarm fatigue. While it is a valuable method for designing alarm management interventions [7], there is no clear association between the number of alarms on an ICU and staff’s subjective alarm fatigue. For example, simply focusing on the number of alarms disregards their temporal distribution (eg, it might fatigue staff more if alarms came in random bursts than if they were evenly spaced out [8]). This could be one of the reasons why in their intervention study, Sowan et al [9] did not find that staff’s alarm fatigue improved despite having managed to significantly reduce the number of alarms on their ICU.
Therefore, we recently developed the Charité Alarm Fatigue Questionnaire (CAFQa), which is a 9-item questionnaire that measures alarm fatigue in nurses and physicians [10]. Using exploratory factor analysis, we identified 2 correlated factors: one revolving around the psychophysiological effects of alarms (eg, headaches and feelings of distraction), and one revolving around ICU staff’s alarm management strategies (eg, customization of alarm limits). We named the former the “alarm stress scale” and the latter the “alarm coping scale.” The alarm coping scale consists of items that are reversely scored. Hence, a high score on either scale is indicative of alarm fatigue.
When developing a new questionnaire, it is essential to establish construct validity, that is, whether the questionnaire truly measures what it attempts to measure. One way to test an instrument’s construct validity is to administer it to a different sample and test whether the originally proposed factor structure reemerges using confirmatory factor analysis [11,12] (for a recent example see Canivez et al [13]).
AimWe aim to validate the exploratively derived factor structure of the CAFQa and thus underpin the instrument’s construct validity.
MethodsEthical ConsiderationsThe ethical approval for this study was granted by the ethics committee of the Charité–Universitätsmedizin Berlin (EA4/218/20) and, if required, confirmed by the local ethics committee at the participating hospital. This study was conducted in compliance with the relevant guidelines and regulations. All participants voluntarily agreed to take part after being fully informed about the study. In study 1, as a reward for completing the questionnaire, we offered participants the chance to enter a draw where they could win a €50 (US $53) voucher for online shopping. Participants were asked to consent to have their data collected, analyzed, and stored anonymously.
ParticipantsIn both studies, we included nurses, physicians, and nurses in training, while excluding other professions and non-ICU staff.
Study 1We recruited participants from 9 ICUs of 5 large German hospitals. The questionnaire was administered on the web using REDCap (Research Electronic Data Capture; Vanderbilt University) between October 2021 and July 2022.
Study 2Using a mailing list, we invited all members of the German Society of Anaesthesiology and Intensive Care Medicine [14] to fill out the web-based questionnaire (again using REDCap) between March 2023 and July 2023.
QuestionnaireThe questionnaire used in both studies was identical and consisted of all 9 items from the CAFQa [10] and 5 general questions about the alarm situation in participants’ ICUs. These general questions were not part of the analysis for this report. All 14 items were pseudorandomly arranged and required responses on a Likert scale ranging from −2 (indicating “I do not agree at all”) to 2 (indicating “I very much agree”). Items with negative valences were reverse scored. Demographic items asked participants about their average number of workdays in an intensive care or monitoring area, their number of years and months of ICU experience, their workplace (campus and unit), and their profession. We made small adjustments to the original wording of 2 items (items 8 and 9) to improve readability: In item 8 we used “situation” instead of “urgency.” In item 9 we used the phrase “clinical pictures” instead of “clinical symptoms.”
Statistical AnalysisAll analyses were conducted in R (version 4.2.1; R Foundation for Statistical Computing) using the following packages: Tidyverse [15], reshape2 [16], psych, semPlot [17], and lavaan [18]. For study 1, we pooled the data from the participating hospitals.
Missing DataIn accordance with Heymans and Eekhout [19], we used predictive mean matching via the mice package [20] to impute missing data that were assumed to be missing at random (MAR). We did not impute questionnaires that were either completely empty or terminated prematurely (presumably due to survey fatigue), as the assumption of MAR was not met in these cases. We assumed that survey fatigue occurred if a participant failed to respond to at least the final 20% of the questionnaire (ie, the last 3 or more of the 9 items of the CAFQa plus the 5 general questions). In total, 0.3% of the data were MAR.
Testing Assumptions of Confirmatory Factor AnalysisIn both studies, the results of the Mardia test indicated that the multivariate skew did not come from a normal distribution with P<.001. Outliers were identified using Mahalanobis distances, with none being detected in study 1 and 4 being detected in study 2 (for both studies: χ29 cutoff=27.9; P<.001). Visual inspection of the data from all 4 cases revealed no unusual response patterns. Given the large sample size, we decided not to remove any outliers. The Kaiser-Meyer-Olkin statistic [21] in study 1 was 0.76, and in study 2 it was 0.8. In both studies, the Bartlett test of sphericity [22] rejected the null hypothesis that the correlation matrix was an identity matrix (study 1: χ236=438.3; P<.001; study 2: χ236=2495.4). There was no evidence of multicollinearity in either study as the determinant of both R matrices was greater than 0.00001 [23] and no correlations were greater than |0.7|. Overall, these results suggest that the data of both studies were suitable for factor analysis.
Confirmatory Factor AnalysisFor both studies, we specified the model in line with our previous findings [10], with 2 correlated latent factors, labeled “alarm stress” and “alarm coping.” Items 1‐5 were assigned to “alarm stress.” Items 5‐9 were assigned to “alarm coping.” Since all CAFQa items are ordered categorical variables (due to being measured on a 5-point Likert scale) and because the Mardia tests indicated that the multivariate skew of both studies did not come from a normal distribution, we used the unweighted least-squares (ULS) algorithm based on polychoric covariances for estimating factor loadings [24-26]. We assessed the goodness-of-fit of the model using χ2, and the following fit indices in line with the cutoff criteria defined by Hu and Bentler [27]: root mean square error of approximation (RMSEA), relative noncentrality index (RNI), Tucker-Lewis index (TLI), standardized root mean squared residual (SRMR), and comparative fit index (CFI).
Convergent ValidityAt the end of the questionnaire in both studies, we provided participants with a brief description of alarm fatigue and asked them to estimate their personal alarm fatigue as a percentage (0% indicating no alarm fatigue and 100% indicating extreme alarm fatigue). We also asked participants to provide their perceived rate of false alarms in their ICU as a percentage (0% indicating no false alarms, 100% indicating no true alarms). To measure convergent validity, we correlated the participants’ mean scores on the questionnaire with the self-provided alarm fatigue and false alarm rate estimations (in total and per factor).
Internal ConsistencyAs a measure of internal consistency, we report Cronbach coefficient α, the McDonald coefficient ω [28], and the mean interitem correlation for both factors.
DiscussionWe aimed to underpin the construct validity of the CAFQa by submitting the exploratively derived factor structure from our previous study to confirmatory factor analysis in 2 independent studies. While the χ2 test rejected the model in both studies, all fit indices indicated a good model fit. The factor loadings ranged from 0.35 to 0.73 in study 1 and from 0.43 to 0.81 in study 2; all were statistically significant. Overall, these results support the hypothesized factor structure. The questionnaire seems to measure the construct of alarm fatigue as proposed in our previous work [10].
The χ2 test is known to be sensitive to large sample sizes [29], which might explain its statistical significance. We did not modify the model because all fit indices indicated a good fit and because model modifications, no matter how small or plausible, can make a model less generalizable.
In both studies, the first factor, that is, the alarm stress scale, and the overall questionnaire were internally consistent. However, the second factor, that is, the alarm coping scale, seems to have issues with its internal consistency. Here, Cronbach α and McDonald ω were 0.49 and 0.55 in study 1, respectively, and 0.55 and 0.59 in study 2, respectively. A similar pattern can be found in our previous study, where the Cronbach α of factor 2 was 0.57 [10]. An internally consistent questionnaire is desirable. However, it can also mean that items are very similar. It was our ambition to create a questionnaire that is brief while measuring the many facets of alarm fatigue. Future studies using the CAFQa should routinely assess the internal consistency of both factors. If the second factor continues to show medium internal consistency, research should be done on how it can be improved (eg, by adding additional items, which would come at the cost of a longer questionnaire).
Because no other measures of alarm fatigue exist that would allow us to estimate the CAFQa’s convergent validity, we asked participants to rate their own alarm fatigue as well as the rate of false alarms they perceived in their daily work. Participants who had a high mean score on the questionnaire also rated themselves as more alarm fatigued (r=0.45 in study 1 and r=53 in study 2). This positive correlation indicates the convergent validity of the questionnaire. Similarly, both studies demonstrated that participants with a high mean score on the questionnaire perceived more alarms to be false in their ICU. In study 1 this association was stronger for factor 2 than for factor 1. This makes sense since a high score on factor 2 (ie, the alarm coping scale) indicates that alarms are not properly managed (eg, by means of patient-specific threshold customizations), which typically leads to more false alarms [30]. However, study 2 could not replicate this pattern. Future research should find an answer to this question: Do ICU staff with a high perceived percentage of false alarms tend to develop stronger alarm fatigue, or do staff that are more alarm fatigued tend to perceive more alarms as being false?
LimitationsThe fit indices RMSEA, CFI, and TLI have been shown to overestimate model fit when using the ULS estimator [31,32], potentially leading researchers to accept a bad-fitting model. Yet, in our case, other fit indices indicated a good model fit. As in our previous work [10], the assumption that participants can accurately reflect and express their own alarm fatigue as a percentage is likely flawed (otherwise, it would not be necessary to develop a questionnaire in the first place). However, most ICU nurses and physicians have heard of alarm fatigue, and we provided them with a brief recapitulation on the concept before having them answer the self-report item in each study. Likewise, it is also probably a flawed assumption that participants can accurately report the rate of false alarms, though Bliss et al [33] showed that participants were able to adapt their response frequencies to alarms based on to the perceived probability that an alarm was not false, thus suggesting that people might have an intuitive grasp of the rate of false alarms in their unit. All in all, we believe that asking these self-rating questions is a valuable method for assessing convergent validity when no other instrument is available.
ConclusionOur results from 2 independent studies underpin the construct validity of the CAFQa. All items consistently loaded onto the factors, as we proposed in a previous publication [10]. When conducting research or quality improvement projects in ICUs, clinical alarm researchers and clinicians can rely on this instrument to measure, compare, and benchmark the alarm fatigue of nurses and physicians.