To test the study’s hypotheses, we used a panel dataset created using 3 major publicly available data sources: COVID-19 nursing home data, Payroll-Based Journal data, and nursing home QM snapshots.

COVID-19 health outcomes for the period May 2020 to June 2023 were obtained from COVID-19 nursing home data. The latter were obtained as a single dataset containing weekly summaries of resident and staff infections and deaths. Payroll-Based Journal data are collected quarterly and offer daily summaries of employee weekly hours for different staff, including nursing, non-nursing, employee, and contract staff. Nursing home QM snapshots include monthly summaries of QMs obtained from multiple data sources, such as the MDS, claims, penalties, provider information, survey summaries, and vaccination data.

The data were aggregated to create a unified dataset. Data aggregation was performed using 2 unique identifiers: one identified the facility (ie, the federal provider number), whereas the other identified the period (ie, week, month, or quarter). All statistical analyses were performed using R version 4.0.0 or higher (R Foundation for Statistical Computing) [31], and a 2-sided P less than .05 was considered statistically significant.

The measures “residents weekly confirmed COVID-19,” “residents weekly COVID-19 deaths,” “staff weekly confirmed COVID-19,” and “staff weekly COVID-19 deaths” monitored the number of residents or staff who tested positive for COVID-19 and those who died due to COVID-19 in a particular week. These figures were reported by providers weekly from May 24, 2020, to June 6, 2023, with the data for the week ending May 25, 2020, potentially including reporting from January 1, 2020, to May 24, 2020.

In the analysis, we combined “residents weekly COVID-19 deaths” and “staff weekly COVID-19 deaths” into a single value representing “total (resident and staff) weekly COVID-19 deaths.” We adopted this approach because both variables were characterized by sparsity and a high number of zeros, and combining them in this manner increased statistical power and, subsequently, the variance for modeling of zero-inflated outcome variables [32].

The set of independent variables was categorized based on the Donabedian SPO framework. According to this framework, the quality of health care, including nursing home care, can be assessed at a facility based on 3 quality components: structure, process, and outcome. Structure refers to the settings where health care is provided (eg, buildings and technology infrastructure) and accessibility features. Process encompasses the actions taken in giving and receiving care, such as pain management, error prevention, and care follow-ups. Outcome indicates the consequences of the provided health care, including mortality rates, readmission rates, and functional status. The quality of health care is contingent upon the interplay of these categories, as Donabedian eloquently stated, “A good structure increases the likelihood of good process, and good process increases the likelihood of good outcomes” [27,33].

For our analysis, we used a comprehensive array of structural QMs encompassing various nursing home attributes (Textbox 1). To gauge the workforce dynamics, we factored in staffing hours, which were calculated separately for employees and contract staff and for nursing and non-nursing staff; we also examined turnover rates for nursing personnel, registered nurses, and non-nursing staff. Furthermore, we considered characteristics tied to the 5-Star Quality Rating System for nursing homes, including facility fines, penalties, and health and fire safety deficiencies, alongside organizational attributes, such as ownership type, provider category, special focus status, and the presence of resident and family councils, all of which contributed to our assessment of overall performance and stability.

An ample set of process QMs was considered (Textbox 2). Among these were factors pertinent to COVID-19, such as the number of residents hospitalized with confirmed cases and their vaccination status, along with weekly admissions of residents previously treated for the virus. The health care aspects included measures such as the percentage of long-stay residents appropriately receiving pneumococcal and influenza vaccines, the percentage of long-stay residents receiving antianxiety or hypnotic medications, those subjected to physical restraints, and those experiencing adverse effects of in-dwelling catheters. For short-stay residents, the metrics focused on appropriate pneumococcal and influenza vaccination rates and the percentage of residents receiving newly prescribed antipsychotic medications.

The outcome QMs used for the analysis encompassed various measures of resident care and well-being (Textbox 3). Long-stay measures focusing on depressive symptoms, weight loss, deteriorating mobility, increased need for assistance with daily activities, urinary tract infections (UTIs), and loss of bowel or bladder control were included. Short-stay measures evaluated functional improvements in mobility, outpatient emergency department visits, and rehospitalizations within 30 days of admission.

Prior to analysis, we removed all the observations in the dataset that contained a flag for either incorrect data or substandard data. This flagging was part of the Centers for Disease Control and Prevention’s quality assurance check, conducted by CMS on 8 data fields to ensure they did not contain implausible or erroneous values. Our decision to exclude flagged observations mirrors those made in prior research in the field [34]. Removing flagged observations eliminated 77,684 observations, resulting in a dataset containing 2,429,283 observations. The latter were used in the analysis after imputing with random forest. The missing value profiles of the variables are shown below (Figures 1-5).

When addressing the presence of missing values in our dataset, we made the deliberate choice to impute rather than exclude them, which aligned with the research goals and the nature of the missing data. In our study, we assumed that the missing data were missing at random (MAR). The MAR mechanism assumes that the probability of missingness depends on the observed data but not on the missing data [35]. As the nursing home data are aggregated at the institution level and include all institutions nationwide, missingness is likely related to differences in administrative or operational factors, such as staffing levels, documentation practices, reporting systems, or governance policies at the state, county, or local levels. We opted for random forest imputation, as it handles various data types seamlessly; it also excels in managing complex interactions and multicollinearity while efficiently scaling to accommodate large datasets. The implementation of random forest in the R package missranger [36] resulted in a reduction of up to 50% of imputation errors, and the average proportional deviation in out-of-bag imputation errors remained within the 10% to 15% range. All the variables (except the outcome variables) were imputed using the random forest algorithm. Although the best method to address MAR missingness is to perform multiple imputation [37], we were unable to adopt this approach due to time and resource constraints. The specifications used for random forest imputation are presented in Table 1.

Data Modeling

Using the imputed dataset, the variable called “percentage of occupied beds” was created using “total number of occupied beds” and “number of all beds.” A total of 10 observations from the calculation resulted in a not-a-number error and were excluded. All the numeric variables were standardized to have a mean of 0 and an SD of 1. All the unordered categorical variables or nominal variables were specified using sum contrasts. For ordered categorical variables or ordinal variables, we used orthogonal polynomial contrasts, and we standardized them onto a uniform scale. Consequently, each contrast column had a mean of 0 and an SD of 1. This approach allowed us to fit the regression models within a standardized framework and easily compare different model coefficients.

Furthermore, all unordered categorical variables containing more than 2 levels were binary encoded before being specified using sum contrasts. Binary encoding is a machine learning technique that combines hash encoding and one-hot encoding. First, all levels of an unordered categorical variable are expressed using a unique numeric value. Then, the numbers are transformed into a unique combination of zeros and ones, making it efficient to store data that exhibit high cardinality.

The outcome variables were characteristic of count data and exhibited properties such as nonnegativity, integer values, positive skewness, and a greater prevalence of lower count values. Equidispersion assessment revealed significant overdispersion, which rendered the use of a traditional Poisson regression model unsuitable. Therefore, we used a negative binomial regression model. Unlike the Poisson model, the negative binomial model accounts for overdispersion with quadratic parameterization. The presence of excess zeros led to the implementation of a zero-inflated negative binomial (ZINB) model, and the panel structure of the dataset resulted in the specification of a mixed effects model. However, specifying a ZINB model requires the presence of zero inflation at all levels of the grouping variables (ie, states, counties, and facilities). Some of the levels did not have zero inflation; they had zero deflation. On the basis of recommendations in the literature [38], the final model we specified was a truncated ZINB mixed effects model.

We estimated multiple truncated ZINB models, specifying different outcome variables. To model the outcomes “residents weekly confirmed COVID-19” and “staff weekly confirmed COVID-19,” the zero inflation (ZI) component of the model included factors influencing initial susceptibility to COVID-19 [39]. These included facility-level characteristics, such as bed size, as well as social determinants of health attributes, including urbanicity, political affiliation of the geographic location, median household income, and local COVID-19 vaccination coverage. To model the outcome “total weekly COVID-19 deaths,” the ZI component of the model included factors influencing the possibility of death after COVID-19 had been contracted, such as lagged weekly resident confirmed COVID-19 cases per 1000 residents and lagged county confirmed COVID-19 cases per 1000 people. We assumed that newly infected residents (ie, infected at 1 time point before) had a higher likelihood of death compared to those who were not newly infected (ie, infected more than one time point before). Our assumption was informed by prior research, indicating that individuals who test positive for COVID-19 typically do not begin to develop detectable antibodies until approximately 1 week after the onset of symptoms [40]. Both lagged variables were calculated for a lag of 1. An overview of the outcome variables and their associated ZI model predictors is presented in (Textbox 4).

The model included random slopes and random intercepts for 3 levels of nesting, that is, nursing home facilities nested within counties nested within states. An autoregressive process of order-1 covariance structure was selected to model the correlations among time points for the same individual. The truncated negative binomial distribution with quadratic parameterization implemented in the R package glmmTMB [41] under the family “truncated_nbinom2” was used.

Mathematically, the model can be expressed as follows:

where subscript I denotes the time point, u denotes the random effects terms in the model, NSZ denotes the event “nonstructural zeros,” P=1−Pr(NSZ) is the ZI probability, and θ is the dispersion parameter specific to the family “nbinom2.”

For each outcome variable, we created 5 base models. Each model contained a different subset of nursing home QMs, namely, outcome QMs, process QMs, and structural QMs (surveys, provider information, and penalties and staffing). Using a subset of significant predictors (30/58) from the 5 base models, we performed principal component analysis (PCA) for dimensionality reduction on the mean-aggregated dataset, where data were averaged across repeated measures for each entity. Using the loadings (eigenvectors) from this PCA, we calculated principal component scores for the complete dataset, including all the repeated measures. Using the PCA scores as predictors, we created 2 predictive models for the states and the components of the 5-Star Quality Rating System of nursing homes. The first 8 principal components explained 80.86% of the variation present in 30 significant predictors (Figure 6).

We performed model fit assessment through a combination of graphical diagnostics, goodness-of-fit statistics, and residual analysis. First, observed versus predicted count distributions were compared using histograms to visually assess the model’s ability to accurately predict both zero and nonzero values. Second, goodness-of-fit statistics were computed, including the log-likelihood, Akaike Information Criterion, Bayesian Information Criterion, and deviance to evaluate and compare across statistical models. We selected several goodness-of-fit measures to achieve a more robust evaluation of model performance. Third, the Vuong test was applied to compare the ZINB model against standard negative binomial and Poisson models to determine whether the inclusion of a ZI component was statistically justified. Finally, residual diagnostics were performed using the R package DHARMa [42], and goodness-of-fit was evaluated through nonparametric, simulation-driven methods. DHARMa provided an assessment of outliers, over- or underdispersion, ZI, and the independence of observations by simulating new response data, computing empirical cumulative density functions, and defining residuals based on observed versus simulated values, ensuring a robust model adequacy assessment.

This study is exempt from the institutional review board, as it involved only the use of publicly available data and did not involve human participants, in accordance with 45 CFR 46.102(e). No private, identifiable information was collected or analyzed.

In this study, we used a panel dataset to evaluate the effectiveness of several nursing home QMs in explaining nursing home QoC in terms of COVID-19 outcomes. Staffing measures (P<.001 for all variables in all models), health deficiency scores (P<.001 for all variables in at least 1 model), COVID-19 hospitalizations (P<.001 for all variables in at least 2 models), and vaccinations (P<.001 for all variables in at least 2 models) exhibited meaningful relationships with the COVID-19 outcomes. The 5-Star Quality Rating System for nursing homes, Medicaid dependency, and ownership showed no clear relationships with the COVID-19 outcomes.

We found a significant association with the percentage of long-stay residents with UTIs. In a retrospective study, the authors evaluated UTI diagnoses and antibiotic prescriptions in 622 COVID-19 hospital ward patients, and they found that 61% of cases had probably been overdiagnosed [43]. Several researchers studying long-term care facilities have also found evidence of overdiagnosis in the form of inappropriate initiation of antimicrobial treatment in 37% to 61% of patients [43-49].

We also identified significant associations with the percentages of long-stay and short-stay residents assessed and appropriately given the seasonal influenza vaccine. A study compared COVID-19 and seasonal influenza patient groups and suggested that preexisting chronic respiratory conditions more strongly impacted the severity of seasonal influenza than that of COVID-19 [50]. It has also been reported that chronic lower respiratory tract diseases claim the third highest number of lives among adults aged >65 years [51-55]. Therefore, older individuals with comorbidities for both respiratory infections may exhibit stronger symptoms of seasonal influenza than COVID-19, demanding management through the administration of seasonal influenza vaccines.

A significant association was uncovered for the percentage of long-stay residents with catheters inserted and left in their bladders. In a recent systematic review of 67 studies, scholars found that catheter prevalence among nursing home residents varied between 2.2% and 36.4%, with a typical rate of 7.3% [56,57]. Catheter-associated UTIs comprise 32% of all health care–associated infections [57,58]. The fact that catheter use is frequently reported in long-term care residents and is a leading cause of UTI helps explain our results, which show significant associations with COVID-19 outcomes for both the percentage of long-stay residents with UTIs and the percentage of long-stay residents with catheters inserted and left in their bladders.

Our findings indicate a significant association between health deficiency variables and resident and staff COVID-19 cases. During the period 2013 to 2017, more than four-fifths of nursing homes in the United States engaged in deficient infection prevention and control practices, including those that substantially reduced the spread of COVID-19, such as proper hand washing and isolation procedures [21,59]. It is possible that these poor practices continued during the pandemic, making nursing home residents and staff more susceptible to COVID-19 infections.

Several scholars have reported that lower resident density, measured by the proxy variable of the percentage of occupied beds, is associated with a lower prevalence of COVID-19 [59-65]. Our study showed a similar association with resident infections but an inverse association with staff infections and total deaths. We found that hospital-based nursing homes reported fewer COVID-19 cases and deaths than nonhospital-based nursing homes, which is consistent with the results of a study by Tarteret et al [66]. Gorges and Konetzka [67] demonstrated facility-level differences in QoC based on the proportion of residents with Medicaid coverage at the facility. Compared to nursing homes accepting Medicaid only, nursing homes accepting Medicare showed clearly better COVID-19 outcomes in only 2 of 3 models. In 1 model, the difference was less clear due to overlapping 95% CIs.

Evidence from various studies suggests that for-profit nursing homes are more prone to COVID-19 outbreaks compared to nonprofit and government nursing homes [59,61,65,68]. However, some scholars have found no statistically significant link in this regard [34,60,66]. We did not notice a clear difference between for-profit, nonprofit, and government nursing homes. Furthermore, no clear difference in outcomes was observed between nursing homes with or without continuing care retirement communities, resident councils, family councils, or resident and family councils. While not a key characteristic, nursing homes newly approved to provide Medicare and Medicaid services had better COVID-19 outcomes compared to long-standing nursing homes. This could be because newer nursing homes underwent inspections more recently than previously approved homes as part of their approval process.

Inadequate nursing staff have been linked to a higher likelihood of COVID-19 outbreaks in some studies [59,62,64,65,68-70], but others indicate no significant connection [61,65,69,71]. Our findings show that increased staffing hours for nursing staff resulted in worse COVID-19 outcomes, whereas increased staffing hours for non-nursing staff resulted in better COVID-19 outcomes. We did not explore these associations further based on the types of nursing and nonnursing staff.

Most researchers have found no meaningful connection between nursing home overall quality rating and COVID-19 outcomes [34,60-62,69,72-79]. Similarly, most scholars have found no clear relationships between COVID-19 outcomes and 2 components of the 5-Star Quality Rating System for Nursing Homes, namely staffing rating and health inspection rating [61,64,70,73,78]. Our analysis did not reveal a clear relationship between the components of the 5-Star Quality Rating System for nursing homes and COVID-19 outcomes.

This study has some limitations. It only considered temporal autocorrelation, without addressing spatial autocorrelation, thus ignoring potential interdependencies across spatial units. Temporal autocorrelation assumes that changes over time are isolated within units, which is rarely the case in interconnected systems, such as health care networks. We also did not perform multiple dataset imputations, which could have resulted in less robust statistical analyses, as a single imputation set may not adequately reflect the variability inherent in missing data. By generating a range of plausible datasets, MIs allow for more accurate and reliable parameter estimates by accounting for the uncertainty around the imputed values. Future scholars should consider modeling both spatial and temporal autocorrelations, as well as performing MIs, to increase the validity and reliability of their findings.

The evolution of quality in the nursing home industry is truly remarkable. However, sustained progress is essential to meet the needs of the growing older adult population. Although widely used in the nursing home industry, the 5-Star Quality Rating System for nursing homes is an unreliable performance measure. As demonstrated through the results of the study, it is hard to delineate nursing homes’ performance based on the star ratings, after accounting for the underlying QMs. Concerted efforts from lawmakers, policy makers, and lobbyists will be required to refine and enhance the measure, thereby ensuring its reliability and effectiveness.