This is technically a link, but I wanted to put it on the main page because it was so good. Our friends at the Blitzburgh Blog have done a nice write up on how to understand DVOA, the principle stat from the footballoutsiders. If DVOA is new to you, check it out.
Their main statistic (the one that gets all the press coverage) is DVOA, or Defense-adjusted Value Over Average. The beauty of this statistic is that it can be used to compare teams as a whole or in any situation you can think of as well as players. Third downs in the fourth quarter? Yep. Second-and-8? Definitely.
Here’s an example of how DVOA is calculated: Suppose the Steelers are facing the Bengals, have a third-and-5 on their own 35, and Rashard Mendenhall rushes for six yards. You place a value on that six yard run, compare it to how other third-and-5’s fare around the league, and adjust for the quality of the Bengals defense.