Yesterday BBC Radio 4 gave Ben Goldacre 40 minutes to broadcast the gospel of the randomised controlled trial, the scientific answer to the problem of deciding which social policies will really work & thereby save the taxpayer squillions of pounds wasted on unproven pet projects of minister & mandarins.
I do not doubt that RCT’s have a role to play in evaluating policy options, but I think the evangelists are underestimating the difficulties & therefore the time & cost of such studies – which, if they cannot produce definitive answers within the 5-year life of a government (or even if they can), cannot bind the incoming government.
There are also problems of defining the unit of randomisation – in education, for example the usual unit would be the teacher, class or school, not the individual child. That makes them more like complex agricultural experiments.
Interestingly when it came to talk about evaluating the teaching of reading there was no mention of attempting a definitive ruling on the question of synthetic phonics versus the rest, results applicable to all children, but just on the best ways of ‘treating’ those who fail to achieve the required fluency in reading in their last year at primary school. This really does seem like a medical model – we are concerned only to treat the minority who are in some way invalid or handicapped
The sense that RCTs apply only to finding treatments for those who, somehow, don’t have the education, nous or sense of civic duty or citizenship to come up to standard, was reinforced by the tale of one of the Downing Street Behavioural Unit’s claimed successes. An RCT found that people who obdurately failed to pay fines imposed upon them by the courts could be persuaded to do so in significantly greater numbers if they were sent reminders by text message rather than traditional brown envelope in the post. The method worked especially well on those who were randomised to receive a text which addressed them by their first name.
The researcher describing this triumph did not say so specifically but there was a sense that this worked because it was more personal & possibly therefore more friendly – certainly more so than threatening them with bailiffs.
But this illustrates the heavy onus that lies on their medical exemplars & on those who take on the responsibility of nudging us into better behaviour: First do no harm. Are you absolutely sure that you are in fact doing good?
'We know your name' is a short step away from 'We know where you live' - a far from friendly phrase to some, one in fact which can be downright terrifying. I was left wondering if the researchers had done any follow-up to find out if the reluctant payers had, in order to find the money to pay their fines, resorted to methods which they would deprecate
Links
Ben Goldacre’s Bad Evidence
