RCTs: the gold standard you say?

Today, the Cabinet Office published a paper on using Randomised Controlled Trials (RCT) for public policy evaluation. The report has been championed by Ben Goldacre, he of Bad Science fame. Its a good report, worth reading (download it here). Except there's one problem: it says nothing of the people involved in RCTs, how they experience them and what that means for their acceptance of any results. This is important - let me explain using the example of the Randomised Badger Culling Trial (RBCT).

A while back I took part in an online discussion for Farmers' Guardian on bovine TB. You can read the transcript and summaries here. I happened to mention that the RBCT should be seen as the "gold standard" in terms of evidence. This was because academics tend to see RCTs as the best kind of evidence possible if you want to prove that something works or not.

I should have known better! If you read the transcripts you can sense Alistair Driver's surprise at what I said, and some of the comments on the debate reflect the same feelings - that the RBCT was not *good* science, never mind the best.

Ive written about this elsewhere, but part of the belief that the RBCT was not good science stems from a lack of trust in those scientists running the trial. There are two things Id point to here. The first is that farmers' own experiences of the trial did not live up to what constituted *good* science. For a start, farmers in some of the proactive culling areas described a war like scenario between protestors and the police which affected the efficiency of the cull. Others described being perplexed at some of the methodologies being used to assess the impacts of the cull on wildlife. There are also some other reasons why farmers rejected the evidence (and other solutions such as vaccination) because of their views and beliefs in nature and ecology.

Secondly, farmers really were not part of the RBCT at all. OK they filled in some 2 hour long questionnaires, and they had to agree for their land to be used - but how involved were they in the experiment? What communication - even - did they receive? A couple of farmers meetings and some newsletters? In fact, the final open meeting of the ISG was held in London - not an area that has much bovine TB - but a location that symbolised the physical and cultural distance between government (science) and the farming industry. Not much attention paid to developing trust there: really, holding it in Exeter would have been better.

Whether or not some of the problems with the cull made any difference to the results (it was supposed to be a real world field trial after all and protests etc are things that happen in the real world) doesnt really matter. The response by the scientists was to say that it didnt, using various calculations and data - the typical deficit response. But by then it was too late and missed the point: farmers - the end beneficiaries of the RCT - had become alienated from it because of the way the RBCT never sought to engage them in it. As I said in the online discussion, for too long farmers have had science done to them, they are not part of the doing of science. And that is the problem.

In a completely different field, Steven Epstein tells a fascinating story of AIDS activists and their gradual incorporation into the running and design of clinical trials for AIDS drugs (paper here but paywall, his book is here). Part of this was down to those activists being able to show how the basic tenets of RCTs (such as randomisation, the use of placeboes etc) were being undermined by the trials' participants. But it also took a lot of scientific learning by these activists to be accepted by the scientific elite. In the end, not only did those scientists running the drugs trials come to value the activists input, but the nature of the trials and the position of the public within them also changed.

So, if we're going to think about doing more RCTs, lets also think about the experiences of being part of one.