Are rosy opinion polls bogus?

A new poll conducted by D3 systems for the BBC and ABC and a German TV station has hit the headlines worldwide. The Globe and Mail says the poll shows "that Afghans are more confident about their future, prefer the administration President Hamid Karzai over the Taliban, and support the presence of American and NATO troops".

However, the poll immediately met with skepticism by knowledgeable and experienced observers like the respected expert on Post-Taliban Afghanistan Daniel Korski as well as journalist Jason Burke. They cited the implausibility of pollsters being able to travel to a high enough percentage of districts to make the sample a valid one. But the criticism goes far beyond that.

The Huffington Post has more:

Afghanistan is a highly patriarchal society, meaning that getting a woman's true opinion is extremely hard. [Prakhar Sharma of the Karzai family-connected Centre for Conflict and Peace Studies in Kabul] said that his research teams have never been able to get even close to the 50-50 male/female split that the ABC survey claims...

Those with experience in Afghanistan were skeptical that the surveyors actually went where they said they did. "If you look at it, the polling was conducted in built-up areas, in urban areas where we have our bases and where the Afghan government has a presence, primarily off the major highways," said [recently-resigned US foreign service official Matthew] Hoh. "So through the South and West of the country, primarily it was done right along Highway 1 where the government has control and where we have control. Off those areas, we don't have control." ...

Hoh said he simply doesn't buy it, both because the areas are impenetrable and because the answers make no sense. "I just don't really believe that, because what I saw in both the East and the South of the country...where all the fighting's really going on, this doesn't jive with, it doesn't agree with what you get when you go out and meet with local villagers. When you go out to these valleys and meet with folks, it doesn't square that they see a central government as a force for good," said Hoh...

Gopal and the others said that Afghan respondents try to figure out what the interviewer wants. "They almost always tell the surveyor what he wants to hear...

"I've seen this first hand when I accompanied surveyors in the field a couple of years ago." ...

Sharma said he has worked directly with the D3 subsidiary and found them to be the best qualified survey contractor among an unimpressive field. But even with that outfit they found "data falsified for insecure provinces (90 respondents in Ghazni had identical responses to all governance related questions, for instance)," Sharma wrote in an e-mail...

"The way the surveys work is by recruiting, say, 34 people for the 34 provinces," [Gopal] writes. "Each of these people are then tasked with finding participants for the survey in their province. In rural Afghanistan, with geographical, logistical and security concerns, these people can't very well go door to door. Moreover, they can't randomly select phone numbers here ... Therefore the surveyors usually find participants by polling their friends and family. This means that you don't have a random sample, and the results of the survey depend entirely on the political outlook of [the] person in charge. Since the surveyors are often educated people who live in urban areas and have ties to the government (in most provincial urban centers, almost every educated person--and there's not many--have family members working for the government, because that's the only job available to them.), there's a heavy pro-government and pro-coalition bias in the surveys." (link)

But easily the most damning criticism of the D3/ACSOR polling comes from Antonio Giustozzi, a British academic who is the recognized authority on the post-2001 Taliban. He was interviewed by British writer Ian Sinclair:

I ask him about the 2009 BBC/ABC News opinion poll conducted in Afghanistan which put support for the Taliban at around 9 per cent and support for the government at 65 per cent.

"The 9 per cent is an underestimate," he replies. Having studied the methodology of previous BBC/ABC polls, Giustozzi explains it is very unlikely that the polling staff travelled to the rural areas in the south - "where the Taliban are" - instead focusing on the cities and provincial centres.

"The sampling is very, very biased ... there are very few unemployed people, whereas even the government says unemployment is 40 per cent.

"In the poll 5 per cent were police and army, whereas in Afghanistan the actual percentage of the population in the army and police is 0.2 per cent. Fourteen per cent were managers and directors. There were no mullahs."

If the sampling was balanced, he estimates the Taliban would get around 15 per cent support nationwide and 30-40 per cent support in the south.

Interestingly, Giustozzi mentions that he has seen polls conducted by the International Security Assistance Force (ISAF) which use different methodologies and get very different results - much higher support for the Taliban and much lower support for the government.

"In ISAF polls in early 2009 support for (Afghan President) Karzai was 4 per cent," he says. "They don't release them, of course - because they show a completely different picture." (link)