Statistical Significance and the War on Christmas

by Kevin Sun

FiveThirtyEight published an article today describing the results of a Public Religion Research Institute survey from two years ago. (

Given recent national uproar over a supposed “War on Christmas,” the article chooses to focus on the choice between “Merry Christmas” and “Happy Holidays” as a winter holiday greeting.

Breaking up the responses by geographical region, the authors find that the region that most prefers “Merry Christmas” is not actually the South as one might expect, but the Midwest where the percentage of respondents who prefer “Merry Christmas” is 11 percentage points higher than that of “Happy Holidays”.


In fact, the South as a whole prefers “Happy Holidays” by a percentage difference of 8%. The article explains that the South’s large African-American population may be one reason for this – Democrats are more likely to prefer “Happy Holidays,” regardless of religiosity.

Also somewhat unexpectedly, the West appears to have a slight preference for “Merry Christmas” over “Happy Holidays” (with a percentage difference of 3%). The article explains that in the less religious states of the West, “Merry Christmas” is less politically charged and therefore attracts less opposition.

But is a 3% difference in favor of “Merry Christmas” even statistically significant? Let’s see.

The graphic states that 1,056 people responded to this survey in 2013. Using Census estimates for the population of the West in 2013 (, and assuming that the survey was indeed “nationally representative,” the number of respondents from the West is just 248 people.

If we simplify by assuming there were no non-responses, a difference of 3% implies that 51.5% of people in the West preferred “Merry Christmas.” The margin error of a 95% confidence interval for this one-proportion Z-test can be calculated as 1.65 times the square-root of 0.515*(1-0.515)/246, which is 5.26%

So the 95% confidence interval for this sample would be 51.5% +/- 5.26%! The true proportion of Westerners who prefer “Merry Christmas” could easily even be *less* than 50%.

Similar calculations show that the 95% confidence interval for the South also contains 50%, whereas the 95% confidence intervals for the Midwest and Northeast do not contain 50%. So only in the Midwest and Northeast can we reject the null hypothesis of “people like ‘Merry Christmas’ and ‘Happy Holidays’ equally” with 95% confidence, and for the South and West the picture is less clear.

So while FiveThirtyEight’s regional breakdown seems to show interesting (and counter-intuitive, and therefore newsworthy) differences between regions, it appears that these survey results – and the somewhat strained attempts to explain them – need to be taken with a grain of salt.





Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s