by Kevin Sun
The College Football Playoff Committee released it’s first Top 25 rankings for the 2015 season last Tuesday, and both FiveThirtyEight and the Upshot decided it was a good time to publish their first analyses and predictions for this season’s playoff.
The two publications take distinct approaches to the problem, revealing unique challenges and limitations of applying statistical methods to a sport as bizarrely organized as top-division American collegiate football, while raising interesting questions about the role of statistics in journalism in general, beyond the world of sports.
(The latest predictions from FiveThirtyEight (with this weekend’s results taken into account) can be found here: http://fivethirtyeight.com/features/heres-how-our-college-football-playoff-predictions-work/ and explanation of their methodology can be found here: http://fivethirtyeight.com/features/heres-how-our-college-football-playoff-predictions-work/, while the latest predictions from the Upshot are here: http://www.nytimes.com/interactive/2015/11/03/upshot/2015-college-football-playoff-scenarios.html.)
As you’d expect, Nate Silver and co.’s way of predicting the CFP is “pretty geeky and statistical,” in their words. The end product of their analysis is a table showing the numerical probabilities of various teams reaching different positions in the rankings, the playoff, and the national title.
One of the first questions this approach raises is quite simply what those probabilities mean. While statistical analysis has greatly improved our understanding of sports such as baseball or basketball, with their high number of discrete and independent events, the sample size of a single football season isn’t large enough for the rule of large numbers to kick in. What does it mean when you say Mississippi has a 75% chance of beating Arkansas, and then they lose? Are you wrong or just unlucky? The next time these two teams meet, next year, the circumstances will be quite different.
The biggest complicating factor also stems from this extremely limited sample size – in a top division consisting of 128 teams, each team only plays against 12 opponents in the regular season. Schedules are unbalanced, so instead of calculating rankings based on win-loss record etc. like almost any other sports league in existence, the rankings are determined by a 13-member committee – and this is another area where FiveThirtyEight’s modeling has not done very well (yet).
Last year, a team which FiveThirtyEight predicted as having a 91% chance of being the playoff – TCU – ending up not making it. So this year, FiveThirtyEight adjusted their model to account for the committee’s tendency to… flip-flop for no obvious reason (paraphrasing). When your model basically has ¯\_(ツ)_/¯ as one of its inputs, you have to wonder if modeling a committee’s decisions with a statistical model is really the best approach.
This brings us to the Upshot’s take on the playoff, which it describes as “A Smarter Way to Look at the Scenarios.” The Upshot tries to really understand the committee’s reasoning, largely by looking at conferences rather than individual teams. But it leaves it at that – no attempt is made to calculate actual probabilities.
Which approach is better? It’s hard to say. The new College Football Playoff system is only in its second year, so FiveThirtyEight’s numerical model could likely improve with more data. The Upshot’s approach seems to make sense, but glosses over the fact that the committee isn’t just one mind, but rather thirteen individuals plus an arcane voting procedure, so the end result doesn’t reflect any member’s thinking.
Looking beyond the sports world, I wonder how often statistical methods really get applied to committee decisions. Much like the Playoff Committee, other decision-making bodies (the Supreme Court, the UN Security Council, company boards, etc.) are obviously constrained by reality and facts, but at the same time there are individual beliefs and agendas, and even procedural idiosyncrasies that can result in broad range of outcomes. And unlike elections, where the rule of large numbers probably simplifies things, committee decisions can much more easily hinge on an individual decision.
As our understanding of sports continues to become more and more quantified, the College Football Playoff Committee serves as an interesting case where individual decisions have an outsized role in determining outcomes, and reminder of the challenges of statistically understanding other, more important, decision-making entities as well.
Because no matter how irrational and historically contingent the structure of modern big-time American college football may be, its quirks pale in comparison to the political and social structures of the “real” world.