What to Do When the Polls Don't Go Your Way

(Welcome, Instapundit readers, and readers from the other blogs kind enough to link to this post. It's a bit on the long side, but I thought the point worth developing in depth. Hope you'll stick around and give the place a once over.)

You've heard about aggressive interrogration techniques "migrating" to Iraq?

Well that ain't all that can migrate, baby.

Here's an article, via Instapundit, on a brand new poll, taken since the elections, showing that Iraqis are more positive about their future than at any point since the invasion. It's not just the greatest spread recorded between those feeling positive and those who aren't, it's also a huge jump since the last poll:

The survey of 1,967 Iraqis was conducted Feb. 27-March 5, after Iraq (news - web sites) held its first free elections in half a century in January. According to the poll, 62% say the country is headed in the right direction and 23% say it is headed in the wrong direction. That is the widest spread recorded in seven polls by the group, says Stuart Krusell, IRI director of operations for Iraq. In September, 45% of Iraqis thought the country was headed in the wrong direction and 42% thought it was headed in the right direction. The IRI is a non-partisan, U.S. taxpayer-funded group that promotes democracy abroad. (My emph.)

Wow. That's pretty positive, huh?

I've written before about how the New York Times handles situations like this. If the scientific polling data cuts against the narrative you're pushing, then it's time for that New York Times specialty: the resturant story! Otherwise known as the "non-poll poll." Or, as I've written before:

When the polls show attitudes that you don't like (that is, attitudes that go against the narrative you've been pushing), not to worry!

Get out there and interview some people -- preferably (when doing a domestic story) in resturants.

Then report (entirely accurately) that people hold different opinions on the matter at hand. Provide great quotes representing both sides. Maybe even provide more quotes from those whose opinions aren't in the majority (if we were to look at scientific polling data.)

Just make sure you never, ever, report on how you decided who to talk to, how many people you actually spoke with, or -- and this is most important of all -- how you chose which quotes to use.

Doing that might make it clear that you're no where close to random sampling and no where close to random selection of quotes, that your quotes were selectively chosen, and couldn't stand up to the polling data. After all, the whole point is to provide people with something vivid and colorful to compare with those cold, impersonal numbers.

So, with yet another positive poll of Iraqis out, it was fairly predictable that there would suddenly be a piece centered on efforts to "sense" the feeling on the Iraqi street appearing in the Times. Just so long as those efforts didn't involve actual scientific polling methods.

Sure enough, today we get, "Many Iraqis Losing Hope That Politics Will Yield Real Change."

The piece starts with the requisite quote from the unhappy single Iraqi, and boy, kudos to the reporter, he really found himself a spectaculary unhappy Iraqi:

Haithm Ali, a wiry blacksmith, was welding an iron gate in his shop in Sadr City, the vast Shiite slum in northeastern Baghdad, when he was asked for his thoughts about the country's new national assembly. Mr. Ali's face broke into a bitter smile.

"I don't expect any government to be formed," he said, his welding glasses pushed up over his forehead. "And they won't find any solutions to the situation we find ourselves in."

What's amazing is what comes next. It displays either a stunning lack of awareness of what's been going on in Iraq for, well, months and months, or is simply a bald faced lie -- and a pretty damn bad one, since even if reporters for the Times don't have Google (which I admit, there are days I begin to suspect that) they have to know that their readers do have it.

He writes:

Nothing like a scientific poll is possible yet in Iraq. But as the national assembly's first brief meeting came and went, broadcast into thousands of Iraqi homes on television, a sampling of street opinion in two Iraqi cities found a widespread dismay and even anger that the elections have not yet translated into a new government. (My emph.)

Wow. Really?

Then how do you explain this? or (pdf) this? or this? this? this? I think I'll stop there.

One could argue there are flaws in these polls. But then they would be flawed scientific polls. Not non-existent scientific polls.

Back to the reporter:

The interviews - which included members of Iraq's major religious and ethnic groups - indicated in particular a striking sense of disillusionment among Shiites, who make up 60 percent of Iraq's population but were brutally suppressed under the rule of Saddam Hussein.

Now, it's entirely possible that those who were most excited about voting are the most angry that a government hasn't been formed. But in the IRI poll, they are also the most positive about the future.

How to reconcile those two? Well, the poll data is fairly fresh, taken up until March 5th. Is it really possible that the Shia have become so frustrated with the extra week's wrangling that they've become disillusioned with the entire process and given up?

Or is it more likely that that's why we put our faith in scientific polling, and not reporters wandering around on the street -- particularly given that those reporters probably aren't wandering around on the street, but are probably sending out stringers, who are likely asking their friends, who are talking about what their friends think, in a circle and network of people who are likely to believe the same thing -- in other words, the problem is the people the reporter heard from are probably self-selected and not random. (Although, hey, this Times reporter gave his stringers credit, you'll notice. Just to be clear.) Because while the poll shows 62% are positive overall, 66% of the Shia in the south are positive.

That is not a "striking sense of disillusionment."

The reporter, besides the blacksmith and a construction worker, a few people walking home, talks to the group of people who show up in one barber shop, for example -- and it doesn't matter how many people show up there. That isn't going to be a random sample.

But, hey, he got some great quotes.

He even talked to an unhappy Kurd.

Of course, according to that apparently non-existing poll, 71% of the Kurds feel good about the future.

But, hey, what are numbers compared to quotes that powerful.

Update: To be clear, the line about Times reporters now crediting stringers refers to this.

Update: Hey! I just remembered something -- forget Google. I remembered a specific article I saw once, and my question is, don't Times reporters have access to an internal data base that archives Times articles?

Update: Strategy Page: The thing of it is, the media doesn't even do a decent job of walking around and getting the feel of the street. (As if this wasn't something we already knew.)

A big story that the media missed was that American troops operating outside the fortified camps (like the Green Zone) were a lot closer to what was going on than your average reporter (who doesn't get out much because of the danger). The combat troops, and many of the non-combat troops, deal with the danger, and Iraqis, on a daily basis. The troops saw the change in attitude among Iraqis. They also saw, in neighborhood after neighborhood, the sharp decline in attacks on coalition and Iraqi forces. They knew the reason for this was the ongoing reconstruction work (mainly supervised by coalition troops) and terror attacks that killed mostly Iraqis. The foreign media appeared to be describing a place that sort of looked like Iraq, but wasn't. Because of the growing availability of email in Iraq, for Iraqis and foreign troops, more people around the world are able to get unfiltered (by journalists) reports from inside Iraq. This has left recipients of these emails wondering what's going on with the reporters. It's simple; fiction always outsells non-fiction.   

March 15, 2005: The Iraqi army and police have a casualty rate over four times that of coalition forces. Amazingly, this has not had any apparent effect on recruiting. When suicide car bombs go off at recruiting centers, more volunteers are lined up in the same place the next day. While most recruits persist because they need a job, for an increasing number, it's all about revenge. Most soldiers are married men who live at home. When police and soldiers are killed, their neighbors in uniform feel an obligation to get revenge. In Sunni Arab areas, the police often know who is doing the killing. If not the individuals, than the family or clan. That's why the terrorists try to haul their dead away. But enough enemy dead and wounded are found, plus captives from raids, to know which families are hostile. The Iraqi police know how to play the family angle, which to Western eyes is bizarre. For example, if it is clear that the family is behind the attackers, then arresting the head of the family (usually an extended family, often with several dozen members) often gets the attention, and often the surrender, of the terrorists.

This comes from Instapundit, but you should read his original post, where a reader emails notes from a talk given by the Commanding General of the 1st Cavalry Division, with all sorts of information you won't hear from the media.

Say, here's a thought: their answer is that the streets are too dangerous, and you have to respect that judgment. No story is worth a life.

Except that there's another option open to them that would allow them to see what American soldiers and Marines see, yes?

There are still all those unfilled embed slots waiting.

Comments

[Dr. Sanity]

You know, Cori, you do a fabulouos job. That was an incredibly good analysis of what is going on. It is extremely frustrating to see these kind of tactics being used by the Times and other MSM outlets over and over again. It is shameless and outrageous. These guys have the audacity to complain about "fair and balanced" Fox. It is either deliberate distortion or complete lack of insight (otherwise known as cluelessness)--or both. Anyway, thank you again for the important job you do. REAL news outlets should hire people like you to keep them honest.

[Richard Aubrey]

There is an interesting result of the emphasis on the use of reserve component troops.
They are spread out throughout the general society, as opposed to the regulars who live on post or nearby in towns whose major demographic is soldiers, veterans, civilian DOD employees, and their families.
When a reservist or Guardsman sends something from Iraq or Aghanistan, he's not telling people who already know, adding another layer, so to speak, to what they already know.
He's telling people out there in the big world. He's spreading it around. Instead of a church congregation in, say, Columbus, GA, sharing the letters of ten guys, we have ten separate churches sharing the letter of one guy each. The news is far more widely spread, as is the comparison to the MSM.
I wonder if the guys who wanted to make more use of the part-timers had this in mind?
I got the general's letter forwarded to me by a guy I've known since jump school in 1970. He remained in the reserves and is still plugged in. He got the letter from a friend and forwarded it to a great many people.
This is my point.

[Stephen Cooper]

Glad you're part of the media mix, Cori. Rant on--the public sphere is the better for it.

[bigR]

RestAurant! With an A. Use your spell check. Jeez!

[Steve D.]

Cori,

Superb post, thanks! I just emailed this to you, then realized I should have put in your article-comments.

I really think you will find this contrast/comparison useful (you need to see the graphics) Majority of Iraqis Are Optimistic- Majority of Americans Pessimistic:

If you want a one-image summary of how dramatic the results are, look at this graph of the trendline for "Is Iraq headed in the right(wrong) direction". Note how significant the improvement is since the poll just before the January 30 election:

Compare the Iraqi trendlines to the American voter trendlines from the latest ABC-WaPo poll. Is there a disconnect here? Is this possibly a consequence of the unwavering media pessimism regarding Iraq as well as WW IV? Is there any correlation with Al Qaeda's Combat Doctrine? See this MEMRI translation "Bin Laden Lieutenant Admits to September 11 and Explains Al-Qa'ida's Combat Doctrine":

... In these wars, the experts stated in their article,[6] 'television news may become a more powerful operational weapon than armored divisions.'...

The reference to the MEMRI dispatch was just emailed to me by counter-terror specialist Dan Darling of Regnum Crucis. Thanks heaps Dan! More to come on this...

[Dan]

Who was this poll taken by again? Oh yeah, the IRI, a "non partisan" outfit funded by US tax dollars. Board members linked below, you decide how "non partisan" they are:

http://www.iri.org/board.asp

Would information like who the chaps are who run the IRI, in your opinion, belong in a fair and balanced news story about this article?

[Miriam]

Those articles always begin: On one sunny (cold, windy, rainy, fill in here), a number of customers who dropped by for their morning coffee (ghat,beer, arak, tea) in a souk (diner, coffeeshop, mall) in Fallujah (Detroit, Grovers corners, Smallville) gave their views on the Iraqi election, (the Chicago Cubs, global warming, whether the moon is made of green cheese) to a visitor....
You can write the rest yourself: the blue collar worker, the teacher, the waitress (all with their ages given) express themselves in homey, folksy, prose, using cute locutions.
The story says whatever you want it to say.
This is journalism?

[Warmongering Lunatic]

I remmeber, thirteen years ago . . .

My hometown newspaper had added a teen-focused section, and it had the results of a presidential poll among teens. This was before the '92 Democratic Convention, and the poll numbers were in line with the national numbers -- Bush in the lead with 40-odd percent, followed by Perot and Clinton. Five teens were quoted in the article -- three supporting Clinton, one supporting Bush, and one supporting Jesse Jackson.

I was outraged by the deliberate effort in the article to not just ignore the poll numbers, but to give the impression of the reverse. Of course, I was 14 at the time. Now I know better than to expect anything different.

[Steve]

Dan.

The names that I recognise in that list do seem to be names that I'd associate with a more conserative point of view. Yes, that sort of information is important - in the same way that it's important to know that a tobacco study has been funded by Phillip Morris, or an environmental impact report has been done under the auspices of the WWF. This information gives you a reason to think that the results of the study might be those that serve the interests of the sponsoring organization, and therefore to justify a scepticism about those results. But it gives you no justification, in itself, to reject those results. The proper course of action for you now would be to look at the study itself to determine whether the methodology and the interpretation are flawed or disputable.

Of course, life is short and statistical analysis is long, so we do often use assumptions of bias as a shorthand; but this is only really justifiable when some organization has a track record of untrustworthiness. The examples I gave earlier will serve here too. I don't think that the IRI has any such reputation.

[Mark]

Total lurker here needing to come out of the woodwork, but be that as it may...

You kick you-know-what, Professor Dauber.

Keep rocking the establishment. Lotta lurkers like me out here love what you're doing... ;)

[puzzled]

This entire rant against the New York Times is based almost entirely on contrasting their recent article, with the IRI poll, and making the assumption a priori that the IRI poll is credible. So, since the NYT article and the IRI poll point in such different directions, that must obviously mean the NYT is being deliberately deceptive to push a political agenda.

Leaving aside for the moment other much more likely explanations for why a New York Times article might be innacurate in the way you assert (for example, do they have google? the other side often asks that same question), should you begin by establishing the credibility of the IRI? After all, you yourself link to a New York Times story from late 2003 that talks about a poll showing Iraqis feeling positively - and yet Iraq was headed in a pretty bad direction at the time, with violence rising sharply over the following year. How accurate was that poll? How meaningful are these IRI polls? Is IRI pushing an ideological agenda? It's certainly at least possible, if not likely.

It's entirely possible that both the IRI poll and the New York Times article are legitimate and real, and present different views of a very large and complex puzzle. They don't necessarily contradict each other. They may be contradictory if your base assumptions are solid: that IRI is doing an unbiased, well run scientific poll, that actually represents how Iraqis feel. You'd do better to tackle that question directly, rather than assume it uncritically.

Even then, it's possible that, for example, sentiment in the country as a whole is different from sentiment in residential Baghdad - Baghdad, after all, has been harder hit by violence than much of the country. And the Shi`ites in Baghdad may have decidedly different views than those in the south, for a lot of reasons I'm sure you're familiar with.

[Joe Yowsa]

Nice article but please spell 'restaurant' correctly next time.

[craig]

Does anyone with more than two neurons firing have any faith in the media these days? They've misrepresented every step of Iraq's transformation, from the war to the occupation and now the post-election period.

Just look at what's happened to all their Iraqi talking points: Najaf, Sadr City, Fallujah. Remember them? Remember how often they were associated with spreading chaos and the disintegration of the country?

How often do you read about them today? Well, you don't -- because they no longer serve the narrative pushed by the gaggle of shallow, lame-brained journalists that pass for the media these days.

[soma_king]

NY Times is appalling. I recall their saying Arafat was getting MORE popular (twice on the headlines within a month), when everyone else was saying he was LOSING popularity. Any doofus reading anything but the NYLiars (i mean TIMES) could tell you that.

But then they had the GALL to claim that Arafat was "recovering" !! From AIDS !! Ha-fuckin-Ha! Arafat was DYING from AIDS. Heck, even Hezbollah knows the Mossad got faggot-lovin Arafat infected with the death virus. Arafat's death wish was written back when the IDF shutoff his compound. After that, he was death warmed over.

Fuckin NY Times. What a pile of dead bodies. Rotten corpses. Die a slow, tortured death Grey Lady. Good riddance.

[Aaron]

http://www.iri.org/countries.asp?id=0000000004

Hey Dems, what's the evil republican slant on this other IRI business?

or

http://www.iri.org/countries.asp?id=1102928473

[dauber]

As to questions about the IRI poll (and, note, it was a poll of the entire country, not just Baghdad) I'm more than happy to entertain the question of whether the poll might in some way be flawed. There are reasons to question any poll, and I myself have in particular questioned how possible it is, for example, for even the best polling companies, with the cleanest reputations, to really do good polls in what amount to police states (Gallup and Zogby's polls of the Arab countries.)

But then you have an argument that scientific polls have flaws, not that they are impossible to conduct (and therefore don't exist) which is what this reporter wrote. He "disappeared" them. That's the first issue; there's no getting around the fact that he flat stated that no polling existed, when there's plenty of it (and if you don't like IRI, he should have at least suggested that the polling from other companies -- surely you have no argument with Zogby or Gallup -- is more dated than his little walking tour.)

The second question that raises is whether having a board you believe to be tipped in a particular direction produces polling methodology that's flawed -- you don't produce any evidence to suggest that's true, and I've never seen anything suggesting anyone distrusts IRI's polls from Iraq. To the contrary, their results run completely consistent with everyone else's, which would suggest they aren't particularly doing anything egregious.

Finally, you are left with the question of whether even a flawed poll is superior to what you get from a reporter and his helpers kind of walking around a bit and talking to a few people, and unless you have evidence that the poll is in someway utterly, fatally flawed, I just don't see how you can't believe that the answer to that question is yes.

Oh -- thanks very much for all the lovely words of encouragement. They mean a great deal. Even if my spelling isn't quite there yet, I'm afraid. (In my defense, I'm just about convinced that a good bit of this is less a spelling problem and more a oh-crap-I-need-reading-glasses problem. That screen's starting to look right blurry now and then. But those of you of a certain age can understand that there's a certain hesitancy in actually finalizing that conclusion.)

[dauber]

I would respectfully ask, however, that you do continue to keep this, ahem, a family blog, if you take my pt.

[Tom]

>>Get out there and interview some people -- preferably (when doing a domestic story) in resturants.<<

>>then it's time for that New York Times specialty: the resturant story!<<

>> he really found himself a spectaculary unhappy Iraqi:<<

Good site; loved the article, but you need an editor. Restaurant not resturant, and
spectacularly not spectaculary. Maybe they're just typos. I do that a lot, too.

[moptop]

Puzzled? I am really puzzled how anyone could give equal weight to a nationwide poll and to a non random selection of a small group of people by a newspaper that has made its negative point of view on Iraq and the administration well known.

I do understand the alure of denial though. I once denied that the majority of people really supported Bill Clinton. It was a dead-end way of thinking, just like the Bush haters are stuck in a dead end now.

[Ten Percent]

Great job. And don't worry about the typos - when Times staffers begin working for you they can be kept busy looking for typos.

[Reason]

Puzzled wrote: "After all, you yourself link to a New York Times story from late 2003 that talks about a poll showing Iraqis feeling positively - and yet Iraq was headed in a pretty bad direction at the time, with violence rising sharply over the following year."

Puzzled, Iraq was heading in a "pretty bad direction" according to whom? To you? Not according to the Iraqis polled. What you seemed most puzzle by is why Iraqis don't agree with you and the constant drum beat of doom from the New York Times. If I had recently been released from the yoke of a dictator, had an interim government and was heading to a democratic election I would have been optimistic too. Yes, even in the face of increased violence, there was hope for a future that could never have been with Saddam still in power. Iraqis understood that.

[IceCold]

Great post. I'm pretty much stuck in the "fortified camp" for now, but the anecdotal indications of continuing momentum for the coalition and a new Iraqi dispensation aren't hard to find among Iraqis. Besides, much as in the US, the biggest factors in the "right direction" question are probably outside the political arena and certainly the interim government/coalition/enemy arena. Potable water and a decent electrical connection on one street in Sadr City would decisively tip the "right direction" result, even though the national direction would not have changed a bit in reality.

[Chuck]

Great post. Don't sweat the typos. Very easy read, very well written. Please keep up the good work.

[internet keno]

[url=http://www.freewebs.com/casino-45/roulette-7.html]casino roulette[/url] http://www.freewebs.com/casino-45/roulette-7.html>online casino roulette http://www.freewebs.com/casino-45/roulette-7.html http://www.freewebs.com/casino-87/keno-1.html>free keno [url=http://www.freewebs.com/casino-87/keno-1.html]free keno[/url] http://www.freewebs.com/casino-87/keno-1.html [url=http://vikimason.com/_disc3/0000083e.htm]no credit check payday loan[/url] http://vikimason.com/_disc3/0000083e.htm http://vikimason.com/_disc3/0000083e.htm>no credit check payday loan http://www.28silverstreet.com/_disc5/00001a52.htm>payday cash advance [url=http://www.28silverstreet.com/_disc5/00001a52.htm]payday cash advance[/url] http://www.28silverstreet.com/_disc5/00001a52.htm [url=http://www.freewebs.com/casino-97/baccarat-5.html]baccarat online[/url] http://www.freewebs.com/casino-97/baccarat-5.html http://www.freewebs.com/casino-97/baccarat-5.html>baccarat online

[free keno]

[url=http://www.freewebs.com/casino-13/keno-2.html]play keno online[/url] http://www.freewebs.com/casino-13/keno-2.html http://www.freewebs.com/casino-13/keno-2.html>video keno [url=http://www.myhedo.com/General/0000006f.htm]instant payday loan[/url] http://www.myhedo.com/General/0000006f.htm http://www.myhedo.com/General/0000006f.htm>instant payday loan http://www.freewebs.com/casino-78/slots-5.html>slots machine game [url=http://www.freewebs.com/casino-78/slots-5.html]slots machine game[/url] http://www.freewebs.com/casino-78/slots-5.html http://www.papillonphotos.com/_disc1/0000173b.htm [url=http://www.papillonphotos.com/_disc1/0000173b.htm]payday advance loan[/url] http://www.papillonphotos.com/_disc1/0000173b.htm>payday advance loan [url=http://www.ispor.org/disc/_disc/00000f10.htm]cash until payday loan[/url] http://www.ispor.org/disc/_disc/00000f10.htm>cash until payday loan http://www.ispor.org/disc/_disc/00000f10.htm