The questions we ask make all the difference

Thanks to Kevin Pearce at the NFU, some of the figures about the badger cull survey and the points that I raised have been clarified - that's great. You can read what Kevin said on his blog - http://kevinpearce.wordpress.com/2011/08/22/public’s-attitude-to-a-badger-cull/, but in brief, the 62% figure was referring to all people who had been asked whether they believed badgers spread TB to cattle, including - and this is the important bit - the people who neither agreed or disagreed. If you add in the people who definitely didn't agree, then the overall support for a cull drops to 52%, but still a majority as Kevin points out.

This raises three questions - all methodological, but one more interesting than the other (perhaps). Skip to the end if you want the interesting stuff and not the lesson in social research methods.

The first is what to do with people who are unsure, or don't know the answer to a question? Do you include them in your analysis? Are people who are unsure (neither agree or disagree) conceptually different from people who don't know? Can you even know that in a short survey? Does it matter anyway? Some interesting questions for methodology geeks, but perhaps with no clear answer, or answers that make much difference anyway? Certainly, it would be interesting to look further at this data. In particular the people who are unsure about badgers passing TB onto cattle: how do they respond to the cull question, are there differences between them in their response in terms of where they live, age, gender etc

Second, what starts to become clear from this analysis are the segments of the population and how their views vary. This is vital for marketing. Clearly from the NFU's survey, those people who agree that badgers pass TB to cattle and are in favour of a cull make up about 30% of the population. And there's about another third who are unsure but agree with a cull and about another third for whom there should never be a cull . It's that middle third - the floaters - where the NFU and others are likely to target their communications. There's no point trying to convince the convinced or the never likely to be convinced. This perhaps is the real value of the survey to the NFU. The analysis not published should give them a breakdown of the kind of people likely to support a cull, and they could if they wanted try to target them, or appeal to their values. It's the sort of thing that politicians do, where the idea of mondeo man and Worcester woman came from.

Finally, and most interesting, is that this analysis now seems to be different from the BBCs survey. The NFU say 62% support a cull, the BBC say the same oppose it. How come there's a difference? Well there's not is there. The difference is in the questions that are asked. I don't actually have the wording of the BBC's survey to hand, and it's a shame they are not more open with the survey data they collect given that it's publicly funded. But the NFU, credit to them, have said what their question was:

Would you be in support of a legal cull of badgers, as part of a range of measures, in specific areas in order to control bovine TB?

This is wholly different to saying do you support a badger cull, yes or no? It's saying in these circumstances do you support a badger cull? And the circumstances help to make agreeing more acceptable. It's the same finding as Defra's citizens panels which concluded that people agree with a cull so long as.........

So the methodological lesson is that you can't compare results from two different surveys unless they ask the same question. This means that it's still not correct to conclude that a majority of the population support a badger cull UNLESS those provisos are included in the headline. This kind of thing happens often, because headline writers want something snappy and not conditional.

Social Research is Good! But the NFU badger cull survey is misleading.

I - of course - like social research, mainly because its my job. But social research can be used to help answer all sorts of interesting questions and contribute to policy decisions. This should be obvious: problems are socially defined - it is society that sets rules about norms of behaviour and what, in general, should happen. Social research is even useful in debates over animal health. In 2007, Defra's Science Advisory Council suggested that Defra needed more social research to help understand issues around behaviour change in all sorts of policy areas, but including animal health and bovine TB - see my paper here.

So, I was pleased to see the NFUs survey of attitudes to badger culling today, but dismayed by the way it represented - see the story here

Here's the reason.

First, the headline in no way matches the findings. Look closely, the press release says that 49% of respondents "agree that bovine TB can be spread to cattle by badgers". It goes on to say that "of those 49 per cent, 62 per cent of people supported a legal cull of badgers in order to control bovine TB". In other words, 30% of respondents were in favour of a cull. This in no way comes close to the claimed "Majority of people back badger cull - survey" headline. Interestingly though it is about the same as the recent BBC survey - see here - which showed that two thirds of the population were against a badger cull. Thats good news - the similar results help provide some legitimacy to the overall findings.

To this the NFU will say but this survey is about looking at the atttudes of people who have somesort of knowledge about TB. And they would point to the fact that of the 49% who agreed that badgers spread TB to cattle 62% agreed with a badger cull. This is fine, but only to a point. You see if you wanted to make that argument then really you would need to ask the respondents a series of questions about their knowledge of TB. As it is, we dont know how may believed in cattle to cattle transmission. It could be that of that 49% percent, half of them didnt believe that cattle to cattle transmission was a problem. If that was the case, then that would also affect the argument that 'knowledgeable' people support a badger cull. But as the survey didnt ask these questions we just dont know. In fact its not clear what question was asked. This is why its always important to do social research properly, or you get misleading answers, or people pointing to 'directions for further research'.

Thirdly, the survey doesnt - from what I can see - go into the circumstances in which a cull is acceptable. People can be in favour of something in principle, but only in certain circumstances or conditions. Added to that is the famous values-action gap: we like to say we would do something, but when it comes down to it we often dont like to go through with it.

Finally, an implicit conclusion to this survey might be: 51% of people are stupid, they need to be educated! Such an attitude is common amongst people who believe in the all pervasive value of 'communication'. In fact studies of risk communication time and again show problems with this deficit model of the public understanding of science. For a start, its only likely that a small proportion of this 51% are going to be open to being persuaded otherwise. Its interesting that the 49% figure is quite similar to Defra research back in the 2000s when members of the public participated in workshops about TB - see here. This stregthened support for a cull, but not by much and even then the participants wanted to put conditions around when it was acceptable (see point 3). Part of the problem then is that these debates are ideological and identity based. Simply shouting knowledge from one side to the other is not going to have much effect. If you never trusted the other side in the first place, you're not going to start now.

So to conclude. I think the NFU should be congratulated on commissioning a piece of social research like this. I think more of it is needed - and not just in relation to bTB. But if its done, it needs to be done properly and headlines need to reflect the findings.