So – in Missouri, take your choice – the Rasmussen poll that has Republican Jim Talent up by 1, or the SurveyUSA poll that has Democrat Claire McCaskill up by 9.
Two polls, conducted by two respected organizations around the same time, two starkly different results. (The SurveyUSA poll shows a 25 point swing among independents from Talent to McCaskill in the span of a month.)
It's possible that McCaskill is up by a few at this moment, and the two polls represent either side of the margin of error. But what are the odds of Survey USA's sample giving McCaskill her best possible positioning, and what are the odds of Rasmussen giving Talent his best possible position?
As I’ve mentioned from time to time, I have a theory that the traditional system of telephone polling is broken. I suspect that caller ID, cell-phone only homes, the general public’s busier lifestyle (who has time to spend a half hour answering a pollster’s questions?), suspicion of pollsters and the media, the spread of push-polling and a host of other factors have come together to make it nearly impossible to get a good representative sample.
(This, of course, presumes all the polls we read are actually conducted. As we saw, one pollster recently pled guilty to making up results.)
I’m hearing I’m not alone in that assessment; apparently in polling circles, there’s some discussion that phone polling is nowhere near as accurate as it used to be. But where I had wondered if this meant samples were underestimating GOP turnout, a respected Smart Washington Guy recently suggested to me that the polls could be wrong in any direction. The Republicans could be doing even worse than the polls show.
UPDATE: Kudos to Constituent Dynamics for showing the world their response rates in the House races they polled. Looks like the response rate ranged from 20.5 percent to 21.4 percent.
This appears lower than the traditional 30 percent or so. (That's measuring who answers the phone.) The cooperation rate - who actually answers the questions - ranged from 28.1 percent to 42.8 percent.
So I'm sure Constituent Dynamics is doing the best they can, putting together the best sampling they can. But according to my calculations, they're still working with data from 5.7 percent to 9.1 percent of the people they call. Maybe it's a representative sample, maybe it isn't.
IMPORTANT UPDATE: Bill Broadhead of Constituent Dynamics writes in, saying that the total response rate should not be calculated by taking the response rate and multiplying that by the cooperation rate. The response rate in the first column indicates Response Rate 1, or the minimum response rate, which is:
the number of complete interviews divided by the number of interviews (complete plus partial) plus the number of non-interviews (refusal and break-off plus non-contacts plus others) plus all cases of unknown eligibility (unknown if housing unit, plus unknown, other).
Over on the other side of the page, the cooperation rates do not represent a percentage of the previous sample. They represent the response rate, with certain categories weeded out, like cases of "non-contact" (phone doesn't connect, I guess?) or where it is unknown if the household is occupied, or "unknown other." For example:
Cooperation Rate 1 (COOP1), or the minimum cooperation rate, is the number of complete interviews divided by the number of interviews (complete plus partial) plus the number of non-interviews that involve the identification of and contact with an eligible respondent (refusal and break-off plus other).
Formulas for both these rates and other rates can be found from the American Association of Public Opinion Research.
I regret the error, which was spurred by the outlandish assumption that the response rate represented the percentage that responded and that the cooperation rate represented the percentage that cooperated.
The good news for pollsters is that my assessment of 5 to 10 percent was too low; the question is, is a 20 to 21 percent response rate enough to get a good sense of the electorate?
Bill also has a bone to pick with the Mystery Pollster's comment that the standard response rate is about 30 percent. Hmmm. Perhaps Constituent Dynamics' response rate is actually one of the better in the business...
