RCTs: the gold standard you say?

Today, the Cabinet Office published a paper on using Randomised Controlled Trials (RCT) for public policy evaluation. The report has been championed by Ben Goldacre, he of Bad Science fame. Its a good report, worth reading (download it here). Except there's one problem: it says nothing of the people involved in RCTs, how they experience them and what that means for their acceptance of any results. This is important - let me explain using the example of the Randomised Badger Culling Trial (RBCT).

A while back I took part in an online discussion for Farmers' Guardian on bovine TB. You can read the transcript and summaries here. I happened to mention that the RBCT should be seen as the "gold standard" in terms of evidence. This was because academics tend to see RCTs as the best kind of evidence possible if you want to prove that something works or not.

I should have known better! If you read the transcripts you can sense Alistair Driver's surprise at what I said, and some of the comments on the debate reflect the same feelings - that the RBCT was not *good* science, never mind the best.

Ive written about this elsewhere, but part of the belief that the RBCT was not good science stems from a lack of trust in those scientists running the trial. There are two things Id point to here. The first is that farmers' own experiences of the trial did not live up to what constituted *good* science. For a start, farmers in some of the proactive culling areas described a war like scenario between protestors and the police which affected the efficiency of the cull. Others described being perplexed at some of the methodologies being used to assess the impacts of the cull on wildlife. There are also some other reasons why farmers rejected the evidence (and other solutions such as vaccination) because of their views and beliefs in nature and ecology.

Secondly, farmers really were not part of the RBCT at all. OK they filled in some 2 hour long questionnaires, and they had to agree for their land to be used - but how involved were they in the experiment? What communication - even - did they receive? A couple of farmers meetings and some newsletters? In fact, the final open meeting of the ISG was held in London - not an area that has much bovine TB - but a location that symbolised the physical and cultural distance between government (science) and the farming industry. Not much attention paid to developing trust there: really, holding it in Exeter would have been better.

Whether or not some of the problems with the cull made any difference to the results (it was supposed to be a real world field trial after all and protests etc are things that happen in the real world) doesnt really matter. The response by the scientists was to say that it didnt, using various calculations and data - the typical deficit response. But by then it was too late and missed the point: farmers - the end beneficiaries of the RCT - had become alienated from it because of the way the RBCT never sought to engage them in it. As I said in the online discussion, for too long farmers have had science done to them, they are not part of the doing of science. And that is the problem.

In a completely different field, Steven Epstein tells a fascinating story of AIDS activists and their gradual incorporation into the running and design of clinical trials for AIDS drugs (paper here but paywall, his book is here). Part of this was down to those activists being able to show how the basic tenets of RCTs (such as randomisation, the use of placeboes etc) were being undermined by the trials' participants. But it also took a lot of scientific learning by these activists to be accepted by the scientific elite. In the end, not only did those scientists running the drugs trials come to value the activists input, but the nature of the trials and the position of the public within them also changed.

So, if we're going to think about doing more RCTs, lets also think about the experiences of being part of one.

Badger culls and the law: some thoughts on the judicial review

The Badger Trust's judicial review of Defra's badger cull plans starts next week (June 25th) and I was asked recently whether there was any judicial review of the original scientific experiment ran by the ISG - after all both the RBCT and the current trial involve killing badgers and are trials (of sorts).

Even though both sides of the badger cull debate now use the ISGs evidence to support their views, when the RBCT was being set up, both sides were pretty much against it. In this paper (apologies, paywall - contact me for a copy) I reviewed the arguments that were being put forward by the National Federation of Badger Groups (now the Badger Trust), the NFU and MAFF. Basically, whilst the NFU were pretty much suggesting that the answer was known already and the RBCT was an academic exercise, the NFBG were arguing that the relationship between cattle and badgers was much more complex and that different kinds of research needed to be undertaken. The NFBG also questioned the statistical design ot the RBCT and arguing that the experiment would not answer some of the research questions that were originally set.

All of these debates took place in the agricultural select committee who paid great attention to TB between 1997-2008. As an aside, its very much a shame that they have not continued to do so. So much for parliamentary scrutiny.

Although the NFBG were against the RBCT, there was no judicial review. Maybe this was because the RBCT was being run as an experimental field trial and that there was no reason to question the powers that MAFF had to do that.

However, the NFBG did refer the RBCT to the Bern Convention - under which badgers are protected by European law. You can read the NFBGs letter to the Convention here and an appendix here

It meant that those involved in designing the experiment had to make the case to the standing committee of the Convention that there would be no local extinction or other breaches.

You can read Maff's response to the Bern Convention here

There is also a commentary on what happened in the ISG's final report (para 2.27) see here

As the ISG's report points out:

In December 1998, while the ISG was still planning the trial procedures, the Standing Committee of the Bern Convention recommended that the trial be postponed pending an opinion on whether it was in breach of the Convention...The ISG assisted and supported MAFF in making the case that, in the light of the severity of the cattle TB problem in the country and the explicitly scientific motivation underlying the planned culling activities, the trial did not breach either the letter or the spirit of the Convention. Twelve months after it had raised the issue, the Standing Committee agreed with this argument and closed its file on the matter, MAFF agreeing to provide it with regular updates both on the RBCT and on the wider TB control programme

I think its fair to say that those involved saw the challenge as something they could have done without. As somebody close to the experiment once told me: it nearly scuppered the whole thing.

In the context of the legal challenge to the current badger cull policy, it will probably be more interesting to see the result of the Humane Society International's complaint to the Bern Convention, rather than the Badger Trust's judicial review. This is partly because of the nature of the current policy and what it could mean for the population of badgers in the UK.

Firstly, the new policy is being framed as a "trial". Now, there are a number of issues relating to the difference between trials, experiments and evidence. Our research is showing that the loose way in which those terms are being used is confusing and misleading for many people involved. Its not just a problem in relation to culling, but vaccination too. But essentially, Defra's argument has been that badger cull methodologies are interchangeable, so this is a trial to test the efficiency of shooting badgers, not its impact. In one sense, Defra could defend the badger cull trials simply as trials - just as the RBCT was. There is no policy yet because there are no results yet: until those results are in, all arguments are hypothetical. Indeed, for Defra ministers to prejudge the results might be better grounds for appeal.

Secondly, if the trial leads to new cull areas being proposed then this would fall into the area of the Bern Convention. One document from Natural England showed that licences for badger culls had been submitted for pretty much the whole of South West England. When the ISG were working on the RBCT they told in no uncertain terms that that prospect was never going to be a realistic policy - not just because it was seen as socially unacceptable, but because of the Bern Convention.

It will be interesting to see what happens at the judicial review. Although the review is framed around 3 issues raised by the Badger Trust, it will be interesting to see whether these matters of the future are matters for consideration, or whether the judges take the view that until the trial is over, there is nothing to rule on. It might be that whatever the result, the legal fight has only just begun.