Lies, damned lies and statistics

The failures of Mid Staffordshire show that mortality data is not the best way to judge a hospital, argues the Patient from Hell Dick Vinegar

What worries me about Mid Staffordshire NHS Trust is that it was not complaints from patients' relatives that triggered the scandal, but the mortality statistics. And when Dr Foster, in 2007, first exposed these rates as being 25% above the national average, the chief executive tried to rubbish them, putting them down to coding errors and poor quality of data. It then took two years before the Health Commission finally exposed Mid Staffordshire in March 2009.

This reminded me of reading, just after I had had a thyroid operation in 2000, that my hospital topped the post-operative mortality league table. So count yourself lucky, dear reader, that I am still here to entertain you.

A year or so later, I found myself making a sick joke about this to an audience of hospital trust chief information officers, in Portcullis House in the heart of Westminster. The joke went down like a lead balloon, and in the coffee break, they all avoided eye-contact with me. Except one, who shuffled over to ask me whether the hospital in question was 'H' hospital. Indeed, it was.

He introduced himself as the IT manager, and explained that the hospital had presented the statistics all wrong. These were the early days of mortality statistics, but he had already learnt the hard way that massaging stats was an essential part of an NHS executive's management skills. The welfare of the patient is secondary.

So, when Richard Francis QC, in his report on Mid Staffordshire in February calls for a single hospital standard mortality ratio (HSMR) as an "impeccably independent and transparent source", he is crying for the moon. Hospital trusts have become expert in manipulating data, and will do so even if the most rigorous mortality standards are imposed, if they sense that their figures are bad.

And, even if they were scrupulously honest and not trying to massage the figures, different coders will interpret the rules differently. That is a law of nature. And the stats will vary between hospital and hospital.

I have an even more fundamental objection to mortality statistics as a way of judging whether a hospital is failing. They are always out of date, even without the delays caused by hospital administrators trying to rubbish them. The original Dr Foster alert in July 2007 covered the years 2003-06. Yet it was not taken seriously until March 2009. During those years, how many people died?

Even in a world with streamlined and accurate HSMRs, by the time they are published they will be at least two years late. That is inevitable. Hence, another law of nature: stats cannot be both timely and reliable.

And mortality stats are not the only criterion by which to judge a failing hospital. Patients may not die because they are left lying in their own faeces. They may not increase the mortality ratio. What matters more, as Richard Francis has twigged, is that the regulators paid no attention to the concerns of patients and relatives, but responded by referring "to data of a very generic type, such as star ratings, CNST (clinical negligence scheme for trusts) levels and so on". He concluded that "benchmarks, comparative trust ratings and foundation status do not in themselves bring to light serious and systemic failings".

Hitherto, statistical data has rooled OK in the NHS. Maybe Mid Staffordshire will be the turning point when the NHS turns away from computerised dodgy stats and tick lists, and turns to the evidence of patient complaints and the evidence of inspectors' eyes, ears and nose.

I apologise to you readers, who are mostly IT types, particularly coders who have sweated long hours producing all this stuff, for degrading the importance of the statistical data. But a lesson in humility never did anyone any harm. Computer systems are not the most important things in healthcare. Clinical skill, sympathy and common sense are.

It worries me though that the six month review of Mid Staffordshire in September 2009 found that it had not yet implemented an effective complaints system. Ingrained bad habits die hard in the NHS.

• Freeing health's data

  Politicians are trying to open access to government data, but this may expose problems with the accuracy of NHS information, reports Michael Cross

• NHS told to exploit business data analysis

  Health organisations should borrow information techniques from retailing, publishing, finance and telecoms, according to the NHS Information Centre

• NHS learns to use its intelligence

  The NHS in England is making increasing use of business intelligence software to plan its services and monitor quality, finds Gill Hitchcock

Close