By Stephan Bisaha
Any poll wonk will point out that looking at just one number in a poll won’t tell you much. The margin of error is necessary to rate the poll and see what it is truly saying. But the margin of error isn’t infallible and they’re even worse in polls.
Take the CBS News 2016 Battleground Tracker, Iowa poll for December 14th through 17th. Ted Cruz looks to be doing pretty well as he has a 9 percentage point lead on Donald Trump. Of course, the margin of error shows that the upper and lower range for each candidate has some cross over so the true numbers can be reversed, but it’s worse than that.
The margin of error is determined largely by the sample size. 1252 registered voters across three states is a great number, but start looking at the questions. They’re broken down by state – meaning the sample size has now shrunk. When we get to question five, “which candidate are you most likely to vote for in the Iowa Republican Presidential caucus in 2016,” – the question everyone is most interested in – our sample is shrunk even further because only Republican primary voters are included. This is a good thing since we’re uninterested in what non primary voters have to say, but we now have a smaller sample size. That original margin of error no longer applies.
While the actual population size of this group is also smaller, the population size does not matter for the margin of error. As the sample each question refers to changes, so to does the margin of error.
When we look back at the crossover between Trump and Cruz, its possible that it is actually much larger.
By visiting the full report, we find the margin of error for Republican and Democrat primary voters – 6.9% for the former and 8.6% for the latter.
CBS was good enough to put the correct margin of error with the graphics in their story about the poll. Real Clear Politics, don’t even bother listing the margin of error without clicking on each individual poll and instead point out how far apart the top candidate is doing against the runner up, despite the previously mentioned uncertainty in that number.
The lesson: Make sure you don’t just check the margin of error, but the right margin of error.