Smart Football has moved!

Please check out the new site, smartfootball.com. All future updates will be made there.

Tuesday, January 06, 2009

Probability, Pragmatism, and Football

I recommend Joe Nocera's recent New York Times Magazine piece on the failures of Wall Street's risk models and overall risk management. It's a worthwhile read of its own right. But I see a football angle.

I've long been a proponent of more rigorous analysis of football, and in particular its associated probabilities. But -- and this is why I am posting this article here -- football is simply so complex that there are limits to our current understanding. With baseball and the "sabermetric" movement, there are attainable gains: the game can be modeled largely as batter versus pitcher, and, within this dynamic, there is only so much change and a finite number of extra variables to control.

With football, however, you have twenty-two players, a variety of formations, receivers, combination blocks, movements, and areas to defend. The game itself follows a dizzying path to completion, with score changes, possessions, downs, field positions, yards to first down, and so on. It's not impossible to model, and there is much we can and have learned, but with football -- as with Wall Street -- what appears certain in your models and statistics may not tell the full story.

"Statistics are like bikinis. What they reveal is suggestive, but what they conceal is vital." - Aaron Levenstein

And, I am sorry to say, that when I read this article I did not think of the excellent work being done that is very useful to coaches and real practitioners, but instead of one group in particular: Football Outsiders. Now, I don't want to be too harsh. They do some great, interesting work, they have illuminated a number of subjects, and, if nothing else, the sheer breadth of what they have addressed will pave the way for more sophisticated analysis. And maybe criticism is unfair, as much of their current stuff is not focused on real football or what might bring real knowledge to the game, but instead on aiding the fantasy football player.

This is understandable, since fantasy football provides ready benchmarks and fantasy footballers are a much broader and more accessible audience than are football coaches. But if we want to be honest about what we're doing, then, to me, we ought to be honest that many of these statistics are for fans and do not actually help anyone (coaches, players, GMs) make decisions.

Football Outsiders' big stat is called DVOA, or Defense-adjusted Value Over Average. It's a nice stat, and certainly it is an improvement over just trying to compare Jimmy and Joe based on their total yards or yards per carry or whatever else. But it's a bit of a fan's statistic, and, further, there's a tendency to adjust it and use it in a way that reinforces what we already know. If DVOA doesn't tell us that Tom Brady and Peyton Manning are the best quarterbacks, then it gets revised until it does, or is rejected until a different measure tells us this unassailable truth. Same with teams: DVOA helpfully tells us that the 2007 Patriots, 1999 Rams, and 1996 Packers were all very good. And, all too often, as Brian of Advanced NFL stats has observed, Football Outsiders far too often "use[s] statistics like a drunkard uses a light post, for support rather than illumination." (Quoting Mark Twain). (One exception: Mike Tanier, though at times his analysis is a bit shaky, is clearly coming into his own and the fact that he has access to NFL film and his practice and diligence is starting to show through.)

Don't get me wrong, I adore statistics and, even more than stats, I think probabilities are the key to understanding almost everything about the world around us. I don't like dealing in absolutes. (Though some seem to vigorously disagree with me when I say that.) If I have to choose between what can only be referred to as the neaderthal view:
"We don't worry about numbers here. Statistics are for losers. I'm not a stat guy. I'm not interested in them, because you can do anything you want with numbers, you can manipulate them, and work around with them. Look at all the financial [problems] we're having in Wall Street right now. That's all those guys lying and playing with numbers. And now all of us are suffering. So I don't believe in numbers, because any crook can play with numbers....It angers me. You know? That's the whole thing, people play with numbers."

- Washington Redskins defensive coordinator Greg Blache

And what I consider the modern view:

"They say statistics are for losers, but losers are usually the ones thinking that. Statistics are great. Our whole game plan is based off statistics. Our management of the game is based off statistics. Our recruiting is based off statistics. Everything we do is analyzed. Is that the bottom line? No. You can't analyze the heart of Tim Tebow."

- Urban Meyer

Then clearly put me in the modern camp. So I really don't mean to denigrate the work of Football Outsiders or anyone else doing progressive work -- we certainly need more of it. Coach Blanche brings up the financial crisis, but it's not like -- even there, where the problems have been widespread -- that numbers and stats can just be dismissed out of hand. They are still useful, but you embrace the limitations: even if your statistics are imperfect, they are often better than nothing, and you must simply recognize and be aware of where the pratfalls are in how they might aid or hinder your decisionmaking. This was lost on the finance world recently, but it applies equally to football decisionmaking.

And, because I also consider myself something of a pragmatist, I nevertheless ask of those doing this work is the same question I would ask of any other idea (because the value of football statistics is simply another permutation of an idea about the value of statistics):
Pragmatism asks its usual question. "Grant an idea or belief to be true," it says, "what concrete difference will its being true make in anyone's actual life? How will the truth be realized? What experiences will be different from those which would obtain if the belief were false? What, in short, is the truth's cash-value in experiential terms?"

- William James, Pragmatism (1907).

So to me, there's a grand opportunity for these statisticians to change the way decisions are made and overall just improve the game -- to make a "concrete difference." Some statisticians have done this, most notably David Romer. His work is so far thorough, practical, and challenging.

I'm not a true statistician or econometrician. And, as I said above, one of the difficulties with doing real, relevant football statistics is that the game is quite complex, and one needs to understand it to model it (and those who understand usually can't model, and those who can model usually don't understand). One of my goals with this site is to try to help bridge this gap.

Nocera's piece begins with this quote from Peter Bernstein's book, Against the Gods: The Remarkable Story of Risk.
The story that I have to tell is marked all the way through by a persistent tension between those who assert that the best decisions are based on quantification and numbers, determined by the patterns of the past, and those who base their decisions on more subjective degrees of belief about the uncertain future. This is a controversy that has never been resolved.

And it will remain controversial -- in finance as well as football -- because the future will be paved by numbers and judgment, marching, somewhat awkwardly, hand in hand.

5 comments:

Ted C is Me said...

Chris: Yet another great piece.

There should, however, be a lot less mystery in the financial world than there is currently:

Austrian Economics

...and in terms of managing risks, the warnings were there more than five years ago, for those able to remove their ideological blinkers:

More Austrian Economics

As with football, economics is often simpler than it appears to the casual observer...

Brad said...

First off I love football statistics and analysis.

But where I think the current where most analyst go wrong is that they are not skeptical enough of their own work.

If practicioners at the highest level do things differently than your numbers suggest then perhaps your theory might be wrong or at least it ought to recieve more scrutiny.

For instance all the statistics say that you should pass more and run less. Yet many coaches believe that the running game is very important. Now it is possible that they are wrong, but it is also possible that the statistics are missing something about the differences between running and passing.

My point is when you incounter a statistic that points in a different direction from conventional wisdom, that should be a clue to re-examine your assumptions. Because their is a lot of experience and survival of the fitest in coaching and the fact that most coaches have coalesed around a certain strategy provides a strong a priori assumption that that strategy is the correct one. If the statistics say otherwise maybe you are correct, but it is probably as equally likely that your analysis is missing some key factor.

This is the same in the finacial markets. If one bond trades at 7% intrest and another trades at 6% and your model says that they have equal risk. You should be very skeptical of your model. Maybe it is an arbitrage oportunity or maybe you are missing some key element of risk (Liquidity, Skewness, Fat Tails, Modeling Uncertainty, etc)

DangerGnat said...

This is my favorite post you have yet made on this website. It lead me on a two-hour trek through my thoughts, reading several of the articles you linked, and also pondering some of the situations in my own life and work that mirrored the ideas presented. I was already familiar with Football Outsiders and will try to digest more of Advanced Football Stats as well, to form my own opinions about the value of each.

I am reminded of the commercial that has been playing recently during football games. I think it's a Nike or Reebok commercial. It shows the players going through practice, expending tremendous effort and enduring injury and exhaustion. All the while, the narrator is talking through the BCS formula, "To get the second component, add the sum of all the computer polls and divide by 1.325", or something along those lines. The game of football is really the clash of the enlightened - statistics, strategy, analysis; and the primal - rage, effort, physical domination. I think it is this dichotomy that has always drawn me to the game, and continues to intrigue me to this day, the reason I can watch a random matchup between two teams I care nothing about, and still get sucked in.

Thanks for the mental exercise today.

Anonymous said...

Hi Chris, I'm not sure if your criticism of FO is very thorough. I think that their stats combine a lot of information and adjust for many statistical shortcomings in the game of football. Although your are correct that they just seem to cite random statistics in their articles, especially some of the weaker ones, I think that they use dvoa, and especially their game charting numbers, to find trends and clues that would simply take too much time and film to identify otherwise.

Anonymous said...

As someone whose education was in economics and law school, I am appalled at the undue influence of the quants in economics and finance. They build huge intellectual structures atop shaky assumptions. They embody hubris on steroids. Perhaps only climate "scientists" are regularly guilty of more certainty supported by less evidence.

As someone who also played 3 sports in college and 1 as a pro and who coached football for a few years in college, I am appalled at the overall lack of interest by coaches in applying analytical tools (of all sorts).

I think there are fundamental problems with much of what FO tries to do, but they are often interesting. The push to try to get baseball-style analysis out of football is simply the square peg in the round hole problem.

Probability analysis is key. Unfortunately, accurate assessment is as much art as science. Most coaches don't understand the probability framework. Most stats geeks don't understad the game.

stan