3.4. Robustness checks
We applied a series of sensitivity tests to our main statistical models, presented in web appendix table 7. The variation in percentage points is presented in web appendix table 8.
First, we restricted the sample to only those Trusts which had one hospital site (63% of the final sample – column 1). The results did not qualitatively differ (0.30 more cases of MRSA bacteraemia per 100,000 bed-days; 95% CI: 0.21 to 0.43). Second we used Coarsened Exact Matching (CEM) to re-estimate our matching models (Iacus et al., 2011), with similar results (0.30; 95% CI: 0.23 to 0.41). Third, to ensure that our results were not driven by the balanced panel, we ran a robustness test including all the Trusts observed at least once, and we find qualitatively similar results. Fourth, we check whether our results were driven by any pre-existing difference between outsourced and in-house Trusts. We replicated our analysis dropping two out of the five years, finding results consistent with our main ones. Fifth, to ensure that our results are not driven by the linear functional form we use a Poisson-model, again finding similar results (0.24, 95% CI: 0.19 0.65). Unfortunately, the models for counting data, such as Poisson models are limited to nonnegative numbers, therefore we cannot compute this robustness check for the log-outcomes. _ENREF_22.
Outsourcing cleaning services was associated with significantly greater MRSA incidence, more reports that handwashing materials are not always available, and patient perceptions of less clean bathrooms and rooms/wards. However, economic costs per bed of outsourcing were also lower.
Our study has several limitations. First, we are currently using data only on Trusts whose MRSA incidence rate was recorded in all five years of the analysis. Attrition might be associated with a higher MRSA incidence rate, although we assume that this is not associated with the cleaning service type. We ran a robustness test including all the Trusts observed at least once, and we find qualitatively similar results. Outsourced Trusts tend to exhibit 0.35 (95 CI: 0.25 to 0.46) more cases of MRSA bacteraemia per 100,000 bed days. In the matching exercise, we were unable to include all Trusts because some lacked data on complexity and only 92 could be matched on these variables. Secondly, we only use data at Trust level, because of the lack of MRSA incidence data at site level. Since different sites within a single Trust might have adopted different cleaning-services, we might have misclassified the type of cleaning service. However, even when we restrict our models to include only single-site Trusts, we find similar results, suggesting that any bias created by misclassification of cleaning services is minor. Third, cleanliness is very likely to affect incidence rates of other hospital acquired infections but MRSA is currently the only infection for which we have comparable data. In addition, MRSA data are limited to infections that are detected in an individual’s bloodstream and not all isolations. Hence our assessment of the problem is likely to be a substantial underestimate. Fourth, we would ideally wish to evaluate Trusts that switched cleaning services; however, in the period for which data were available, relatively few trusts switch, and a complicating factor is that these switches were likely to have occurred in relation to performance issues. However we can draw on the findings of a study that introduced an extra cleaner to two matched wards for six months each, using a crossover design, and found a 27% reduction in infections with MRSA, with the benefit disappearing after removal of the cleaner (S. J. Dancer et al., 2009). This is directly relevant to our finding that outsourced cleaning employs fewer staff. Fifth, we do not have any information on the screening practises used by the Trusts but there is no reason to believe that this would be systematically different between the in-house and the outsourced ones. Sixth, we did not have any data on staff-turnover or recruitment and/or sickness leave, which might be a good measure of both job-dissatisfaction and cleaning quality. Seventh, using data from several years before our study, we found no evidence that those Trusts outsourcing cleaning were systematically less clean, a possible cause of confounding by indication. However, caution is required as we cannot be sure that the Healthcare Commission data exclude a selection effect. Unfortunately, there are no other data that would be able to do so.
These findings have important implications. Although, from a narrow accounting perspective, Trusts outsourcing cleaning seem to incur lower costs of cleaning per bed, this is also associated with fewer staff and reduced reported availability of hand-washing material as well as an overall increased incidence of MRSA. However, it is not possible to conduct a full economic analysis because of an absence of comprehensive data on the nature and severity of the entire range of infections associated with poor cleaning, any additional deaths, the additional cost of treatment, and any associated costs, such as litigation. This is clearly an area for future research.
Notwithstanding these limitations, the fact that the antibiotic armamentarium is rapidly depleting means that our findings should be considered a reason for considerable concern.