Why Eric Grenier Got the Canadian Election Wrong

by Christopher Huffaker

On Monday, October 19th, Canada had a parliamentary election that surprised basically everyone. The three party tie of August had narrowed to the Liberals and the Conservatives by the homestretch, and the Liberals had even taken a moderate lead, but nobody expected a Liberal majority government. That includes Éric Grenier, the man who styles himself as Canada’s Nate Silver (his site, 308, is named for the number of ridings that existed when he created it. There are now 338). His Poll Tracker on the CBC website projected a liberal government, but with closer to 146 seats. They got 184.

Polls at left seats at right.

At left: Grenier’s poll averages from September 10th through the election. Red is Liberal, dark blue Conservative, and orange NDP. Light blue, green, and grey are Bloc Quebecois, Green, and Other, respectively. At right: Grenier’s final seat projections. Colors are the same as in the polling aggregates.

184 is, in fact, within the range of values he provided, barely. His maximum value was 185, as the 95th percentile of his projections for the liberal seat count. I asked him why he used 95th for the “max” rather than 100, given that this would theoretically make his projection wrong 1 in 20 times. “The polls use that degree of confidence in their estimations,” he told me, so it seemed a natural measure to use.” Fair enough. However, he wrote for the CBC, those ranges are “designed to take into account major errors in the polls. The polls had no such errors.” As he pointed out elsewhere, the only extent to which the polls were wrong was that support was shifting rapidly toward the liberals in the final days, and they did not poll on the day of the election. In popular support terms, they were almost exactly right. The problem for the projections was in the way that popular support became seats.

            I had been checking the poll tracker every day for a couple of months, and the morning of the election I wrote the following to my professors:

“At its core, Grenier’s projection methodology is poll aggregation, with some added factors like incumbency, “star candidates,” floor-crossings, and leaders (who tend to outperform their parties when their parties do poorly). The highest weighted polls are the most recent ones, with the largest sample size and best methodology (phone calls), from experienced pollsters with good histories of results. During the election, the weight of each poll is reduced 35% each day after it was released.

“In 17 federal, provincial, and municipal elections, ThreeHundredEight.com’s vote projection model has outperformed the average error of the final polls conducted by all pollsters during a campaign 15 times and has, on average, had an error level of 2.18 points per party compared to 2.7 points per party for the polls.

“The projection is complicated by the fact that the PM is elected not at a province-wide level, like the President and the senate in the U.S., but by the plurality of ridings, more like the House. This presents a difficulty because there are very few riding-specific polls, so generally Grenier is reduced to using data at a provincial level while factoring in the other information mentioned above. Once province-wide vote projections are in place, he uses what he calls a “proportional swing method” to project the number of seats; if they’re polling at r times their vote in the last election, he projects they’ll get r times the number of seats. (He says that in the UK a “uniform swing method” is used instead.) It works well when tested with past results, he says.”

This swing model turns out to be what killed Grenier’s projections, and in fact most everyone else’s, too. The unpredictable thing in this election was that there were a ton of voters who didn’t vote in the last election, and they mostly voted for the Liberals. These people were distributed in ways disproportionate to previous elections.  That is, Liberal voters weren’t all where Liberal voters usually are.

For example, the Liberal Party outperformed everyone’s expectations in B.C. significantly. He explains, “four-fifths of Liberal gains can be attributed to new voters. On average, the Liberals earned just more than 16,000 new votes in their B.C. gains over their 2011 performance in the same ridings. The Conservatives lost an average of 3,300 votes in these ridings, and the New Democrats just over 800 votes.” As Grenier complained to me, there is a “relatively small amount of data available in Canada about voters.” Without having riding-level data in British Columbia, he made his best guess based on past Liberal performance in the province, and maybe with past BC voters he was correct. But he couldn’t make a relative projection for new voters; maybe the UK uniform swing method would have made more sense, for them.

Grenier's projection for BC (above) versus the actual seat distribution (below).
Grenier’s projection for BC (above) versus the actual seat distribution (below).

That point, he points out, is likely to be moot in the next election; the Liberals promised to do away with the first-past-the-post (riding-level plurality) election method, so in 2019 he’ll need a whole new model. He told me that right now he uses a simple spreadsheet for his projections, but he “may move to something different and more sophisticated in the future.” Perhaps a whole new electoral system will provide the necessary motivation.

NB: Grenier doesn’t make his model open source, as the New York Times did with Leo, but the methodology he lays out on his website is pretty comprehensive. He doesn’t have a background in statistics, and his work is for a mass audience, so basically he’s just trying to “make things easier to understand for people bombarded with polls.” Why making his model available would hurt that, I don’t know, but I suppose it’s his livelihood.