Please visit the new home of Majikthise at bigthink.com/blogs/focal-point.

« Hamdan lawyer pushed out of military | Main | Weird news »

October 16, 2006

More on the Burnham/Lancet study of Iraq deaths

Andrew of Statistical Modeling, Causal Inference and Social Science: Estimating Iraq deaths using survey sampling. [HT: Chris of Mixing Memory.]

Alon Levy of Abstract Nonsense: Responding to the Iraq Body Count Distortions.

And, Another Lancet Roundup from Tim Lambert of Deltoid.

TrackBack

TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8341c61e653ef00d834f24bb269e2

Listed below are links to weblogs that reference More on the Burnham/Lancet study of Iraq deaths:

Comments

Lindsay,

I don't know of anyone on the web who has done a better, more responsible job keeping the public informed on the development of this story than you. Thanks so much!

Maybe the warbloggers could get some tips on disputing figures from David Irving.

Iraq Body Count is extremely skeptical of the Lancet's extrapolations.

I think the quickest way to see what's going on is to go to Table 4 in the original paper and multiply everything by 2000. Among other things, the implication is that over 70000 people have been killed by car bombs since 2003. It is hard to claim that high-mortality car bombings would have gone unnoticed in such quantities.

It's hard to claim that, but when you have evidence, there's not much you can do.

Since when are surveys "evidence". Idiot liberals.

Wishful thinking doesn't make it so. You know, like when liberal guys tell everyone their penis is at least 5". You can hide behind your wishful thinking, but it doesn't change reality.

From a real scientist: For those who can read with an open mind

money quote:

When I pointed out these numbers to Dr. Roberts, he said that the appendices were written by a student and should be ignored. Which led me to wonder what other sections of the survey should be ignored.

With so few cluster points, it is highly unlikely the Johns Hopkins survey is representative of the population in Iraq. However, there is a definitive method of establishing if it is. Recording the gender, age, education and other demographic characteristics of the respondents allows a researcher to compare his survey results to a known demographic instrument, such as a census.

Dr. Roberts said that his team's surveyors did not ask demographic questions. I was so surprised to hear this that I emailed him later in the day to ask a second time if his team asked demographic questions and compared the results to the 1997 Iraqi census. Dr. Roberts replied that he had not even looked at the Iraqi census.

And so, while the gender and the age of the deceased were recorded in the 2006 Johns Hopkins study, nobody, according to Dr. Roberts, recorded demographic information for the living survey respondents. This would be the first survey I have looked at in my 15 years of looking that did not ask demographic questions of its respondents. But don't take my word for it--try using Google to find a survey that does not ask demographic questions.

And there's more.


Oh yea, sweet wet dreams, liberal losers

So, a commenter has linked to the rebuttal by IBC.

http://www.iraqbodycount.org/press/pr14.php

Those neocon, fascist, Bush apologists, citing other sampling studies.

Fat chance of seeing something that contradicts Lindsay's party line in a post here.

Dale, you do know who Steven E Moore the author of the Opinion Journal piece is, right?

Moore worked for Paul Bremer of the CPA as a public opinion consultant. So, he's got a bit of an interest in public opinion regarding the slaughter of Iraqis that took place partly under his boss's watch.

His objection is bogus. He's saying that there weren't enough cluster points. What counts as "enough"? Well, whatever he says. Ideally, you want the smallest margin of error possible. The margin of error of the Burnham study reflects the relatively small number of cluster points. Nobody says they made any mistakes in the calculation.

Moore is spouting bafflegab about "demographic questions." He's saying that if the researchers had asked demographic questions, we could ascertain whether the sample was "representative" of Iraq. Bullshit. The researchers did collect info on the ages and genders of the people who were said to have died and the number of people in the household.

Those neocon, fascist, Bush apologists, citing other sampling studies.

The post of mine that this very post of Lindsay's links to addresses that other sampling study.

Lindsay said, "The researchers did collect info on the ages and genders of the people who were said to have died and the number of people in the household."

Lindsay, the idea is not to just "collect" the data but to use it to validate the sample by determining if the sample is representative of the population ("universe" in stat jargon). Neither study did that.

That's the major point of both Moore and of Apfelroth (guy from Albert Einstein College Medicine writing in Lancet).

There are minor wrinkles:

1. The ages and genders of the dead aren't the desired info, but the ages, genders (marital status, education, etc) of those alive in the household.

2. Whether the study collected age and gender is ambiguous.

The study itself says, "The
survey listed current household members by sex, and
asked who had lived in this household on January 1, 2002.
The interviewers then asked about births, deaths, and
in-migration and out-migration"

But Moore's WSJ article says, "Dr. Roberts said that his team's surveyors did not ask demographic questions. I was so surprised to hear this that I emailed him later in the day to ask a second time if his team asked demographic questions and compared the results to the 1997 Iraqi census. Dr. Roberts replied that he had not even looked at the Iraqi census. And so, while the gender and the age of the deceased were recorded in the 2006 Johns Hopkins study, nobody, according to Dr. Roberts, recorded demographic information for the living survey respondents. This would be the first survey I have looked at in my 15 years of looking that did not ask demographic questions of its respondents. But don't take my word for it--try using Google to find a survey that does not ask demographic questions."

http://opinionjournal.com/editorial/feature.html?id=110009108

Maybe Moore and Roberts both meant non-recording of more detailed demog data like education, marital status, ethnic sub-group, etc.

But there's a MUCH bigger problem. The study says: "The third stage consisted of random
selection of a main street within the administrative unit from a list of all main streets. A residential street was then randomly selected from a list of residential streets crossing the main street. On the residential street, houses were numbered and a start household was randomly
selected. From this start household, the team proceeded to the adjacent residence until 40 households were surveyed."

So:

1. That method excludes all streets except those connected directly to a main street. This is highly non-random and in fact invalidates the sample.

2. Bombs tend to be located in or near main streets, so the "main street effect" combined with street selection technique could well have yielded a sample biased toward violent incidents.

For the record, I loathe Bush and his arrogant insistence upon a catastrophic invasion when so many (foreign and domestic) advised him against it for many reasons. So I hardly support the invasion. But I honestly think the Hopkins study still poorly-done. A repeat of the UN/ILCS survey (using its superior stat methods) asking more general violent-death questions would be ideal.

Gnerally, in any such controversial scientific matter, I think "who said what" is much less important than examining the science ab inizio -- but it's hard to ignore that Nobels have assailed the Lancet's addiction to headline-seeking over good science:
http://www.timesonline.co.uk/article/0,,2-1658807,00.html

The comments to this entry are closed.