HSMR dead and buried?

HSMR dead and buried?

The use of HSMRs for assessing the quality of healthcare or even comparing mortality rates has been much questioned in the academic community. Dr Foster is a commercial concern, who naturally have an interest in selling their product. The Academy of Medical Royal Colleges (AoMRC) have no such conflict of interest, and they published their report on international HSMRs yesterday.

The AoMRC have been looking at the comparison made by Professor Sir Brian Jarman between UK and US healthcare, where he stated there was a 45% difference in mortality. That figure was striking, and gave rise to a number of salacious headlines. A difference of that magnitude based on such a large database would be beyond the vagaries of mere statistical chance. 

However, the AoMRC noted a number of factors that made any direct comparison unreliable:

Choice of factors included in the risk adjustment model

Choice of data used to derive the model

Choice of hospitals included from each country

Choice of assuming the same level of underlying risk in each country

There were also a number of differences in coding which affected comparisons:

Differences in diagnostic practices

Differences in the number of secondary diagnoses

Differences in severity

Proportion of population deaths in acute hospitals

Their conclusion?

"We do not believe that the comparisons are valid because of the reasons we have given. More rigorous analyses using more clinically sophisticated data need to be used. This reflects the conclusion of a recent paper from the Dr Foster Unit which stated: ‘There are considerable challenges in combining administrative databases across countries, with decisions to be made regarding definitions of inpatient admissions and diagnosis and procedure groups. Interpreting the resulting case mix-adjusted in-hospital outcome rates across international boundaries is hampered by differing discharge policies and intermediate care facilities. Standard statistical models cannot take account of this without valid and complete post discharge information’.

One of the key issues is that we have no measure of the uncertainty attached to the estimate of 45%. On a simplistic level, it is quite accurate because it is based on large numbers, but uncertainty in almost all the key assumptions used in its derivation mean that we cannot have much credence that this estimate is even close to the actual value.

We would advocate comparisons of outcomes of care for specific conditions or interventions

based on high quality clinical data that enables adequate risk adjustment. Such comparisons are difficult as they require data from the two countries that are sufficiently similar as regards variable definitions, patient inclusion criteria, similar professional judgement regarding diagnostic labelling etc. Despite these challenges we would recommend that some studies be commissioned to establish if there are systematic differences in the quality of hospital care between the two countries."

So how and why did this study with such precarious findings become headline news? The usual reasons I suspect. An eye-catching headline, a chance to bash the NHS, and further validation of the utility of HSMR. Only that might just have backfired.