Political Polling: Who Can be Trusted?

Blog Post - Political Polling: Who Can be Trusted?

Ever wonder about the validity of election polls? Which firms can be trusted? Which ones are biased? Which ones have performed most accurately over time? And what does that say about today's most recent polls? Perhaps the simplest measure of poll accuracy is to measure how close its last predictions came to the actual voting results. For instance, if a poll said the Republican would win by 5 points and he ended up winning by 1 point, that would be a 4 point error. Gather thousands of such examples together and we'd be able to assess the quality of pollsters. Luckily, such a database exists at fivethirtyeight.com.

"The database includes nearly every poll conducted in the last three weeks of a presidential, U.S. Senate, U.S. House or gubernatorial campaign since 1998, along with polls in the final three weeks of presidential primaries and caucuses since 2000." [a] In addition to measuring the simple error rate, a bias towards political parties can also be calculated using a "mean-reverted bias." That's defined as "a pollster’s historical average statistical bias toward Democratic or Republican candidates, reverted to a mean of zero based on the number of polls in the database." [b] In other words, if all the polls from a hypothetical 2012 election - averaged - predicted the Democrat winning by 6%, but he only won by 4%, then the average bias of all polls was +2% in favor of the Democrat. To look for a political bias in individual polls, then, one must treat a 2% error rate as "the new zero" since EVERY poll was off by about that much. That's the distinction between a simple error rate and an observation of bias. One figure is a literal measurement of how wrong a pollster has been whereas the other is how wrong it has been relative to how wrong ALL polls were. This database has tabulated results from several thousand polls, with the raw data available via spreadsheet download for all to freely scrutinize. [c]

We cannot list every result, since there are hundreds of them, but are the results for 20 of the top polling firms [d], in order from smallest error rate to largest:

ABC/Washington-P: 3.0% (Dem bias of +0.6%)
Pew Research: 3.4% (Dem bias of +0.6%)
Fox News/Opinion: 3.9% (Rep bias of +0.5%)
NBC/Wall Street: 4.2% (Dem bias of +0.5%)
Reuters/Ipsos: 4.3% (Dem bias of +0.1%)
Gallup: 4.3% (Rep bias of +0.8%)
SurveyUSA: 4.6% (Dem bias of +0.1%)
CBS/NY Times: 4.7% (Dem bias of +0.6%)
Quinnipiac: 4.7% (Rep bias of +0.7%)
PPP: 4.9% (Rep bias of +0.2%)
USA Today/Suffolk: 5.2% (Dem bias of +0.7%)
Mason-Dixon: 5.2% (Rep bias of +1.0%)
Rasmussen: 5.3% (Rep bias of +2.0%)
McClatchy/Marist: 5.4% (Rep bias of +0.7%)
CNN/Opinion: 5.5% (Rep bias of +0.1%)
Zogby: 5.6% (Rep bias of +0.8%)
Gravis: 5.8% (Rep bias of +1.1%)
Fox News/Anderson: 6.4% (Dem bias of +0.4%)
Economist/YouGov: 6.7% (Dem bias of +1.6%)
American Research: 7.6% (Rep bias of +0.1%)

The notable observation is that, despite these being the most reputable pollsters, NONE of them average an error rate less than 3%, most appear to have an error rate of 4-5%, and some even have an error rate as high as 7.6%. Recall that these are NOT outliers. These are results from hundreds and hundreds of previous polls. Keep this in mind when reviewing current presidential polls. Use it as a guide. For instance, the most recent polls listed on RealClearPolitics, with polling ending on 7/12/16, 7/13/16, 7/11/16, and 7/9/16, are from CBS/NYT (tie), Rasmussen (Trump +7), the Economist (Clinton +2), and McClatchy (Clinton +3). [e] However, since we know CBS/NYT has an average error of 4.7%, Trump could actually be ahead or behind by 4.7% per that poll, despite it reporting a tie. Rasmussen has a 5.3% error rate and is often biased in favor of the Republican, so maybe Trump is actually hovering around a 2 point lead per that poll, despite it reporting a 7 point one. The Economist has a whopping 6.7% error rate and is often biased in favor of the Democrat, so the fact that it reports Clinton having a 2 point lead means almost nothing. She COULD be losing by 4.7%, for instance, which happens to match one of the possibilities shown in the CBS/NYT poll. And lastly, the McClatchy/Marist poll has a historical error rate of 5.4%, so it showing Clinton with a 3 point lead might actually mean she's behind by about 2%.

 

CONCLUSION:
A reasonable assessment of the pollsters shows they're generally off by around 4-5%, regardless of political bias, and that the leading presidential candidates are likely locked in a near tie (presently), despite some polls implying that they're further apart.
--------------------------
Sources:
[a]
http://fivethirtyeight.com/…/how-fivethirtyeight-calculate…/

[b]
https://github.com/…/data/blob/m…/pollster-ratings/readme.md

[c]
https://raw.githubusercontent.com/…/pollster-stats-full.xlsx

[d]
http://projects.fivethirtyeight.com/pollster-ratings/

[e]
http://www.realclearpolitics.com/…/general_election_trump_v…

 

Comment list

    Post has no comments.

Leave a Reply

Your email address will not be published. Required fields are marked *

Captcha Image

Blog Search

Newsletter

Subscribe to our newsletter