Misleading Data Visualisation: Defra’s TB Maps

UPDATE: Since I wrote this, Defra have been in touch to say that they have removed the maps from their website and are changing the data included on them, which is good. The challenge of how to present these data remains though.

Maps are powerful tools. They can be a great way of presenting (or visualising as people like to call it these days) data and statistics.  But as geographers know, maps are not always what they seem – and these maps (link no longer working) from Defra are a surprisingly good example of that.

It all looks nice and shows the advance of TB across the country – that part may be true. But actually, the maps are misleading. Here’s why:

The maps are supposed to represent the number of cases of TB. Only, the asterix tells us that it’s actually only about confirmed cases of TB – that’s cases where visible lesions have been found. In the new even more confusing jargon, these are “OTF withdrawn” cases. Or is it?

Below the picture you can see the number of reactors. Only, this is the total number of reactors – not the number of confirmed reactors (OTFW). Its a bit of a muddle – what is the map actually showing? If it’s about confirmed cases then why not give the actual number of confirmed TB reactors? That number would actually be a lot lower than the figures presented. I don’t have the figures of confirmed reactors (i.e. reactors with visible lesions) to hand, but for herd breakdowns figures show that around half are OTFW – although this varies by region: in Dyfed OTFW breakdowns account for 36% of all breakdowns, whilst in Gloucestershire its around 60% - quite an impressive difference considering both are supposed to be high incidence areas.

Personally, I think this whole debate about confirmed/unconfirmed reactors is not handled very well. Sometimes Government likes to treat unconfirmed reactors in the same way as confirmed cases (as is the case here), all go under TB restriction, and even when no visible lesions are found, Government likes to tell farmers that doesn’t mean they don’t have TB. At other times, they are treated separately – for example, the work on badger culling ignores unconfirmed cases, AHVLA only visit farms to conduct epidemiological investigation on confirmed cases, and the rules for going clear are different. If one of the purposes of the change to the OTF jargon was to help farmers understand the difference between confirmed and unconfirmed cases, then I think that misses the point about the reasons why farmers distrust science and the government.

Whatever the figures are, the shading on the maps is also confusing. I’m bound to point this out as a geographer –the maps are committing a gross ecological fallacy. Some of the later maps (2006, 2010) are essentially suggesting that every farm in the south west is under TB restrictions. We know this is not the case: even in the high incidence areas the number of breakdown farms is relatively low: Defra’s own stats show that for last year the % of herds with a confirmed (OTFW) breakdown is around 9% in Devon, 12% in Gloucestershire, and 4% in Dyfed. When you look at the numbers of farms under TB restrictions it’s higher, but not by too much: around 19% in Devon, 28% in Gloucestershire and 14% in Dyfed. So to suggest that all farms in these areas have TB as these maps do is misleading. But the maps also fail to distinguish between high/low incidence areas of TB. For example, looking closely you might think that an area like South Wales (low incidence) has the same kind of TB problem as West Wales, where the problem is much greater.

None of this is to say of course that TB in these areas isn’t a problem or needs to be reduced. It may also seem to be a bit of pedantry – those “in the know” understand the limitations of presenting the data in these ways. But this is the point: these maps aren’t meant for people in the know: they are an exercise in communicating science to people “not in the know”. And the main reason why you’d want to do that is to justify your actions to those people – to convince them that what you are doing is right.  So they are not simply objective representations: they have a purpose. Normally, in cases like these, I’d say something like this is a good example of why people end up not trusting the Government. However, there are similarities here with the use of statistics by other organisations involved in TB: the NFUs public attitude survey and the Badger Trust’s use of TB statistics being other examples. And when you take all three cases together you can see there’s a wider purpose to the use of TB statistics, found also in many other studies of policy making. Simply, it’s to repeat a well known fact in policy studies: statistics help to objectify claims, frame debate and advance the interests of different organisations.

TB in Wales: what is really going on?

The other week the Badger Trust put out a press release about TB statistics in Wales - you can see it here (pdf). Ive been meaning to comment on it. Its not that its all wrong: its just that the picture is more complicated, and potentially more interesting than they let on. And the whole story doesnt completely support their argument either.

The Badger Trust's basic argument is that TB is declining massively in Wales, especially in Dyfed where a badger cull could take place. They say that this decline means that any badger cull should be off the agenda. This is right - see the graph below: the numbers of TB reactors in Wales have declined year on year since 2008. They are still quite a lot higher than they were in the year before foot and mouth disease which is responsible for the big spike after 2001. And they are also a lot higher than in the early 90s.

But the problem with statistics is that on their own these overall numbers don't really tell use the whole picture. Medical statistics use all sorts of methods to standardise mortality rates or disease incidence across populations taking into account age profiles in different regions. Even then you have to be careful about interpreting differences between different regions - they may just be occurring by chance.

TB stats arent treated in the same way, but one thing we can do is to compare the overall amount of disease with the actual number of cattle tested. This relative figure is important as it tells us the actual incidence of disease and this is a better indicator of whether TB is going up or down. You can see this in this graph. There's little point comparing the number of reactors between different time periods if the number of cattle tested is quite different.

Reactors per 1000 cattle tested (RpT) is used as a standard indicator to show how TB levels have changed over time. You can see that for Dyfed and Wales in this graph. This graph is a bit like the other one. It shows that there has been a decline in RpT since 2008, but its still twice as high as pre-FMD, but more than half the extremely high rate of 2008. It also suggests - provisionally - that the decline may have ended. The 2011 figures are only for the first 6 months so should really be treated with a lot of caution, but if the trend continues for the next 6 months, then the decline may have ended.

Perhaps its no surprise not to see the Badger Trust commenting on this: their point is that the overall trend is still downward from 2009. Describing a new upward trend would spoil their story: although at least they do for England. Here though they are basing their data on 2010 figures. Its all getting a bit confusing: using a trend from 2009-2011 for one area and 2010-11 for another. Using stats to make a point? Maybe thats what statistics create...

There's an interesting story just looking at reactors from routine tests - that is tests where a reactor is found putting farms under TB restrictions, rather than tests designed to get disease out of an already infected heard (i.e. short interval tests). I dont have 2011 data for routine tests, so what the most recent trend is I dont know. But what is interesting is that these data suggest that, proportionately, the number of reactors is now getting back to similar levels to before FMD.

Just for good measure, here are the figures for new breakdowns from routine figures. Just to be clear, these are the % of tests of clear herds that result in either a reactor, or a reactor and/or IR. The first graph is for Dyfed, the second for England and Wales. Again the trend is similar: that its taken 10 years to more or less get back to where we were before foot and mouth disease. In fact for England, the trend is probably down from that before FMD.

So what do all these figures mean? On their own these statistics cant really be used to justify support for anything. They dont really suggest that additional cattle controls in Dyfed are having an effect: for a start the area whey they are being applied is just a very small part of the county, and then we'd need to look at other similar areas to see what was going on in them.  Actually this would be a really useful thing to do and to make those figures publicly available. There's potentially a whole load of other interesting stats too: the number of confirmed breakdowns; the number of repeat breakdowns; the survival times of herds between breakdowns etc etc. Wouldnt it be good to show all these data too? After all, arent we all supposed to be "armchair auditors" these days? Hmm. The criticism of armchair auditing is that sometimes statistics need to be handled with care, even when they are presented in relatively simple terms. In fact, changing the way Defra present TB statistics was one of the recommendations of the ISG.

Do these statistics support the Trust's claim that they undermine a badger cull? The basis of a badger cull has not really been made in terms of the level of disease. That's to say no-one has said above a certain level of disease then we need to start thinking about a badger cull, and below a certain level then its not required either. Remember, when the Krebs review was commissioned it was thought that there was a problem with TB and the government needed to look at a badger cull, but those figures were a lot lower than now. So even if the decline continues it wont change that, although it would affect the cost-benefit analysis for a badger cull.

Perhaps this disconnect between the need for badger culling and the level of TB is the most interesting thing about these statistics. Maybe what it highlights is the difference in expertise involved in disease control: the difference between a field approach which relies on local grounded experience to get the job done, and one which relies on more a scientific quantitative account of disease. In many ways thats what the debate in TB is about.







Note on data
You can recreate these graphs using data held on Defra's TB statistics website. To calculate Rpt its: no. reactors divided by number of cattle tested multiplied by 1000. For some reason in recent years they've stopped giving the actual number of cattle tested which means you cant calculate RpT. The data Ive used comes from an extract from Vetnet that Ive been doing research on. Ive not included any data from pre-movement tests. The 2011 figures are from the latest TB statistics release.