There’s more than one way to Get the Margin of Error Wrong

By Stephan Bisaha

Any poll wonk will point out that looking at just one number in a poll won’t tell you much.  The margin of error is necessary to rate the poll and see what it is truly saying.  But the margin of error isn’t infallible and they’re even worse in polls.

Take the CBS News 2016 Battleground Tracker, Iowa poll for December 14th through 17th.  Ted Cruz looks to be doing pretty well as he has a 9 percentage point lead on Donald Trump.  Of course, the margin of error shows that the upper and lower range for each candidate has some cross over so the true numbers can be reversed, but it’s worse than that.

The margin of error is determined largely by the sample size.  1252 registered voters across three states is a great number, but start looking at the questions.  They’re broken down by state – meaning the sample size has now shrunk.  When we get to question five, “which candidate are you most likely to vote for in the Iowa Republican Presidential caucus in 2016,” – the question everyone is most interested in – our sample is shrunk even further because only Republican primary voters are included.  This is a good thing since we’re uninterested in what non primary voters have to say, but we now have a smaller sample size.  That original margin of error no longer applies.

While the actual population size of this group is also smaller, the population size does not matter for the margin of error.  As the sample each question refers to changes, so to does the margin of error.

When we look back at the crossover between Trump and Cruz, its possible that it is actually much larger.

By visiting the full report, we find the margin of error for Republican and Democrat primary voters – 6.9% for the former and 8.6% for the latter.

CBS was good enough to put the correct margin of error with the graphics in their story about the poll. Real Clear Politics, don’t even bother listing the margin of error without clicking on each individual poll  and instead point out how far apart the top candidate is doing against the runner up, despite the previously mentioned uncertainty in that number.

The lesson: Make sure you don’t just check the margin of error, but the right margin of error.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s