Saturday, January 08, 2011

The limits of statistics

Three interesting reports have come to my attention this week, all dealing with the perils, problems & pitfalls of incorporating science, via mathematical statistical models, into the policy making process.

The first is 'A Statistical Analysis Of Multiple Temperature Proxies: Are Reconstructions Of Surface Temperatures Over The Last 1000 Years Reliable?' to be published in Annals of Applied Statistics. The short answer, given by authors Blakeley McShane of Northwestern University and Abraham Wyner of the Wharton University of Pennsylvania, is ‘No’. But the paper has 'discussion' status & comments are being published simultaneously. It is a serious contribution to the debate, & it will be fascinating to see how, if at all, the paper is reported in the wider media & on the BBC.

As someone who concentrated on the often underrated business of data collection I can but cheer editor Michael L. Stein’s remark that data are more important than models and models are more important than specific modes of inference.

Data problems loom large in the story of the fallout from what are now seen to be hopelessly wrong estimates of the number of swine flu deaths to be expected during last year’s epidemic. David Spiegelhalter had a Thunderer column in The Times about this & there is a version with links to original documents on his website under the rather alarming title Stalin had a point.

The paper on Lessons Learnt is especially instructive. One impression I gain from this is that there was a distinct lack of clarity in the organisation, responsibilities & reporting lines in the Department of Health – what a surprise from New Labour.

I was diverted by the conclusion that it is all about managing expectations, & especially by the idea of producing a general epidemiological primer for ministers & senior officials which would, among other things explain ‘how early analysis can be impacted by stochasticity’. However the report ends with a humble recognition that experts too need a much better understanding of the types of questions that ministers and senior officials need to ask (& answer for & to press & public) during an epidemic while making decisions in conditions of huge uncertainty.

The third report also concerns problems over what are said to be inadequate collections of official data on the effects of vaccination & came in a programme on Radio 4, The Vaccine Casebook. As the BBC’s own webpage rather excitedly puts it “Richard Phinney reports from the West African country of Guinea Bissau, where a team of Danish and African medical sleuths have pieced together evidence that could change public health care forever.”

The work, led by Dr Peter Aaby, consists mainly in the keeping & analysis of long term health records of children, has been reported extensively in the specialist press & illustrates the very real problems in this area. It may have detected harmful effects, it may even have established that gender really does matter, even in babies, when it comes to differential responses to medical prophylactic treatments, but for the moment there seem to be only observational studies. Meanwhile international health experts must, surely, continue with what may in the end turn out to be well meant interventions which did a deal of harm.

We are not in an MMR situation but some good people may end up with a loss of reputation at the very least. Either someone will end up as the visionary hero, others the stubborn deniers of evidence which challenges their cherished assumptions, or it will all prove to be a misguided but understandable misinterpretation of evidence caused by overenthusiastic adoption of a novel hypothesis.





Posted by at
Labels: