Mocking the Colts: Football Outsiders DVOA Projections for Colts Absurd

Aaron Schatz and the staff at Football Outsiders put a great deal of effort into developing advanced statistics for discussing NFL football. Coltzilla certainly appreciates their efforts to provide fans with another perspective in evaluating football play, comparing players or teams to one another, and getting a deeper look into functional performance.

There are times stories at Coltzilla will reference those numbers in an effort to illustrate or support an observation or argument. Moreover, Jacob Crocker has spent a great deal of time recording and developing his own statistics to help evaluate the play of individual players on the Colts roster in 2010.

That said, I am not a huge fan of statistics – advanced or otherwise – because they are devoid of any real context more often than not. Applying advanced statistics to the football field is complex and often unreliable because the football field includes so many variables that is not readily recordable without some kind of subjective interpretation.  Proponents of football statistics often trumpet their innate objectivity as one of the biggest reasons they are so valuable. The reality is that while numbers on a page are objective, no interpretation of those numbers ever is, and the accuracy or intrinsic value of those numbers cannot always be validated.

All of this is important because Football Outsiders released their 2011 DVOA Projections on Tuesday. In those projections, the Colts are 8-8 — with a healthy Peyton Manning.

In order to evaluate their method, one first needs to understand what DVOA is and how it works. From the FO website:

The majority of the ratings featured on are based on DVOA, or Defense-adjusted Value Over Average. DVOA breaks down every single play of the NFL season to see how much success offensive players achieved in each specific situation compared to the league average in that situation, adjusted for the strength of the opponent.

[A more in-depth description] This sounds nice, and a system that could accurately record and apply these statistics would be a really great tool for people to use when they’re placing bets or if they want to enter prediction-based fantasy league. The problem with these kinds of statistics, particularly in a projection scenario, is the number of variables included to reach a projection.

The funny thing about statistics is that the more data points you use to create a number within a relatively small variance (16 games, 32 teams, 200 yards passing versus 300), the more those numbers are going to tend to revert to the mean. In English? Really good teams will be projected with fewer wins than they probably will get and really bad teams will be projected with more wins than they will probably get. The teams in the middle will gravitate toward the middle.

I picked a year where I thought there could be a real example of this, just to check my theory. In 2007 the New England Patriots went 16-0 and the Miami Dolphins went 1-15. How did the Football Outsiders DVOA projections think the season would go for each respective team?

The 2007 projection gave the Patriots 12 wins and the Dolphins 6 wins. It was off by four and five wins respectively. DVOA predicted nine wins for the Colts, after a 12-win 2006 season and Super Bowl championship. The Colts were 13-3 in 2007. The average teams who were predicted in the middle of pack in terms of wins were typically closer. Some of those middling predictions were correct.

How do these correct predictions occur? Honestly, the biggest reason is coincidence. There are 16 games for any NFL team, making the possible variance between prediction and result no greater than 16. If a prediction is off only by a couple of wins, it doesn’t seem like a big deal.

Example: DVOA predicts the Colts go 8-8 even with Peyton Manning in 2011. If the Colts went 10-6 with him, it would be easy for people to just shrug off the two game difference and think, “the DVOA prediction was pretty close.” Sure, it sounds nice. Off two games? Not bad.

What if the variance levels were higher and the units DVOA was predicting increased significantly? What if DVOA was trying to predict something that had 16 million units to work with? A two game difference all of the sudden becomes a two-million unit difference. A statistical prediction that inaccurate wouldn’t even be relevant.

A miss that big is like using the metric system when the English system of measurement was supposed to be used. Sure, it’s the same percentage miss because it is all relative, but it would be so many units off it would be tossed out — or be a $125 million embarrassment.

I provide an extreme example to illustrate the point. There is little doubt in my mind that there is value in the statistics Football Outsiders gather and report. There is also little doubt in my mind that DVOA has value in terms of trying to reflect how a team or individual has performed in a given game, or even season. As a “backward looking” illustration tool, DVOA and other advanced statistics are at least relevant — though they still create a blind environment because it would take watching and understanding each individual play to know the significance of how the numbers were generated.

I say this because I don’t want Aaron Schatz, Football Outsiders, or any of Coltzilla’s readers to think that I am trying to simply dismiss all of the work people like Schatz, or even Jacob Crocker do. I value the work they do, appreciate it, and have used it on numerous occasions to illustrate or support my own observations and arguments.

Still, statistics will not be able to accurately predict Donald Brown rushing for 129 yards and a touchdown. Statisticians will call those kinds of performances “outliers.” Funny though, those outliers seem to find ways of happening, and they alter team records, individual performances, and the outcome of statistics based predictions like the one Football Outsiders produces every year.

Statistics lack the human element. They won’t be able to accurately predict that the 2007 Giants will defeat the undefeated Patriots in the Super Bowl. All traditional and advanced statistics would generate a conclusion that the Patriots were the superior team. (and maybe they were)

Statistics cannot predict, nor apparently take into account, the significance of player injuries, returns from injury, drafting rookies, signing free agents — general team roster changes and improvements or losses. They won’t be able to accurately gauge player development. These things, these individual performances, these human things that are not represented by numbers, they happen. They’re important. Heck, sometimes they determine everything.

So while I realize that Football Outsiders, other sources for advanced statistics, and those who make projections based upon them, do so without theorizing and without being “subjective” in their minds. It doesn’t change the fact that, at the end of the day, the admission of those facts makes it patently clear that those predictions and projections ultimately are nothing more than throwing darts.

The numbers make us feel good, like we have some reason to believe what we believe about what will happens. It’s a nice security blanket for predictions, because at least then we tell ourselves we’re not just “going on our own whims.” But it doesn’t work that way.

The fact is, predicting the Colts go 8-8 with a schedule that — no matter what the Football Outsiders numbers suggest — is favorable compared to 2010, is ridiculous. If Manning and the Colts entered the season now and even cut the 2010 injuries and missed games in half, 10-6 would be an absolute floor for a realistic prediction. If Football Outsiders formulas don’t reflect that, they don’t work.

Are human predictions “superior” than the ones that Football Outsiders put out? Well there’s a fun question because it depends on just as many variables as the statistics do. In fact, entirely unquantifiable variable like bias, team knowledge, etc. will come into the equation. Still, if one is informed enough about the current status of each team, how and why those teams match-up the way they do, have knowledge about field conditions, weather, etc., there is no reason to believe that even without statistics an informed person can’t be just as accurate.

Coltzilla will go into more detail breaking down the schedule and showing where the Colts should be favorites to win with and without Manning in another story. For a preview of that breakdown, visit Mocking the Colts from Tuesday.