A primer on EPA, from Aaron Jones to Aaron Rodgers

From an advanced statistics perspective, this was unquestionably the year of EPA (Expected Points Added). While advanced stats in football have been around in some form for about 15 years now (really starting with Football Outsiders and DVOA, which to its credit, still holds up pretty well), there hasn’t been a major advance that’s cracked the mainstream in quite some time. EPA looks poised to do just that, and in many ways it already has.

If you’re unfamiliar with EPA, or haven’t bothered to learn it because some Twitter personalities use it to slag Aaron Rodgers every day, this post is for you, because EPA is actually very simple to understand (even if it’s extremely difficult to calculate). EPA is generally available in real time and don’t be surprised if it begins to pop up on broadcasts in the not too distant future.

So what is EPA, when is it useful, and what do you have to watch out for? Let’s start with this post about Aaron Jones from Bill Barnwell of ESPN. The Packers, as it turns out, are much much more efficient with Aaron Jones on the field than with Jamaal Williams. How much more efficient? Well...

The Packers have run 603 offensive snaps with him on the field. Those snaps have added 90.4 expected points to Green Bay’s offensive output, or 0.15 expected points per play. Only two offenses in football generated more than 0.15 EPA per play this season -- the Ravens and Chiefs. With Jones on the field, the Packers are the third-most dominant offense in football.

Without Jones, they aren’t the same. When Jamaal Williams has replaced Jones in the lineup, the Packers have generated minus-24.33 EPA across 337 snaps, or minus-0.07 EPA per play. The Williams version of the Green Bay offense ranks 29th in the NFL. Aaron Rodgers’ QBR drops from 67.7 with Jones on the field to 28.7 with Williams in the lineup instead.

To calculate EPA you simply take the number of points a team was likely to score before a play was run versus the number of points a team is expected to score after the play was run. Subtract A from B and you have the expected points added. Those expected points numbers are based on models that account for down, distance, and yard line, and statisticians then compare the models’ predictions against actual outcomes in order to verify effectiveness. EPA is very dependent on the era, and if football becomes higher-scoring or lower-scoring for whatever reason, the existing formula won’t work quite as well. Using the modern EPA formula for a game from the 1980s, for instance, won’t work because the scoring environment was lower.

If you look at the quote above from Bill Barnwell, it seems to make the case that Aaron Jones’ mere presence makes the Packer offense one of the best in the league, averaging about a sixth of a point per play, while Williams is a huge drag. But here, we must be careful. While I do not doubt that the offense is better with Jones overall, EPA can sometimes struggle with painting an accurate picture of rotational players like running backs or defensive players.

The problem is that Jones plays the most on first and second downs, and indeed, the Packers are among the best in the league on first and second down. First and second down are also not as impactful as third down. Think of it like this; If you fail on first down, you still have second and third down to fix it. Your situation hasn’t declined that much. If you fail on 3rd down though, your situation, and therefore your expected points, are MUCH worse, as you will in many instances be punting or forgoing a potential touchdown for a field goal attempt.

By DVOA, Williams is the better receiving back of the two, even if he is less explosive, and Williams is also slightly better in pass protection. (Jones, it should be mentioned, is no slouch in either category, and is more explosive as a receiver.) But Williams also plays more often on tougher downs, where conversions are not particularly likely. Every 3rd and 11 that isn’t converted with Williams on the field dings his EPA many times more than a similar play on first and ten, and because Williams sees a lot of plays like this, he looks worse than he really is.

Ben Baldwin of The Athletic actually talked about this phenomenon recently in regards to Jadeveon Clowney:

EPA is a great measure of team efficiency, and it works quite well for quarterbacks as they play every offensive snap. But, as with any stat, context is important and it’s easy to use it wrong.

What has EPA taught us?

EPA has confirmed much of what we thought we knew. For instance, the relative value of passing over rushing, generally speaking, is not in dispute. It allows us to gauge fourth down decisions with more precision, and using stats like EPA per play allows us to accurately compare teams and players with different sample sizes in attempts. EPA correlates very closely to actual performance, and because it can be as granular as one play, it’s very flexible in how it can be used.

Perhaps the most surprising result is the relative value of turnovers. While no one doubts that interceptions are a bad thing for a quarterback, we now have a decent measure as to how they stack up versus other forms of offensive failure. Punts are, after all, turnovers, and leading an offense that punts too much can be as bad as leading one that’s careless with the ball.

Baldwin ranks quarterbacks using a composite metric of Completion Percentage over Expected (CPOE) versus EPA. You can read a primer on CPOE here, but it essentially measures whether a QB is completing more or fewer passes than we would expect based on depth of target and level of competition. CPOE is one of the more useful metrics in predicting success from the NCAA to NFL level, and it was very high on Russell Wilson and Kyler Murray. CPOE is also a nice stat for contextualizing different types of quarterbacks. If a QB is a great check-down artist, completing say, 85% of passes where we would only expect 70%, he will score well. Similarly, if we have a real gunslinger hucking it deep all the time and completing 58% where we would normally expect something like 50%, that QB will also rank well. By combining CPOE with EPA, which simply measures efficiency per play, we can also tell whether a QB’s style is working well, and this is where it gets interesting.

Below you will see two charts: A composite score of CPOE and EPA, and a graph of CPOE v. EPA. You’ll notice that Aaron Rodgers sits close to the middle, and you will also notice he has an interesting QB with him in Tampa Bay’s Jameis Winston.

Rodgers and Winston could not be more different. Rodgers is extremely conservative with the ball and rarely throws interceptions (just 4 this season), while Winston is notoriously careless, and threw 30 picks this season. But, there is a cost to Rodgers’ conservatism as the Packer QB averaged just 7 yards per attempt while Winston averaged 8.2. Outside of his picks, Winston was prolific, while Rodgers was more of a game manager.

Winston has received a great deal of scorn for his interceptions (and wonderment at one of the biggest boom or bust seasons of all time), but Rodgers should, according to EPA, see almost as much for his conservative nature. EPA sees the good plays that Rodgers isn’t making as well as the bad ones Winston makes, and on the whole, they are pretty much equivalent.

This may sound insane. Surely, the guy with 30 picks is much worse than the guy with 4 regardless of the other passes they threw. I thought the same thing, and so I dug a bit deeper. First, the Packers may not have thrown as many picks, but they punted a ton more:

• Packers: 177 drives, 77 punts (43.5%)
• Buccaneers: 198 drives, 57 punts (29%)

Had the Packers run 198 drives like the Bucs did while maintaining the same percentage for drives ending in punts, they would have punted 86 times, or 29 more than Tampa. Given that the difference between Winston and Rodgers was 26 interceptions, you can see how it’s maybe a closer call than you thought.

That said, punts are generally not as damaging as picks. While it’s never good to give the ball to an opponent, you at least get to tack 45 yards on, which is helpful. That said, not all picks are created equal, and some are relatively benign. Winston threw plenty that were devastating, including a game-losing pick-six on his final pass of the year, but two of his picks were also Hail Marys, which are about as benign as you can get. Winston also had a tendency to throw picks at the very beginning of the game, leaving plenty of time for the Bucs to make up for the mistake and catch up. Finally, Winston also took plenty of chances in “negative EPA situations.” 13 of his 28 non-Hail Mary picks were in 3rd and long (or worse) situations that required some risk to move the chains and maintain possession. Those are not necessarily “good” interceptions, but Winston’s willingness to take risks there also pays off in more first downs, and fewer punts.

If you’re still skeptical of Winston’s value versus Rodgers — and again, I was too — I actually charted EPA’s prediction of the value lost by Winston’s interceptions against what actually transpired in the game. The goal was to see if his turnovers, in real life and not just the model, cost the Bucs more than the model predicted. In reality, EPA was much harsher on Winston than real life (by about 9 points through week 16), as Buccaneers opponents had an almost uncanny tendency to turn the ball back over immediately. Winston threw a ton of pick-sixes this season, but EPA is great at measuring those as the play ends in an opposing TD, and there is no guess work as to how much they scored. At least in 2019, EPA was too hard on Jameis Winston. Cementing the analysis, on a per-drive basis, the Packers averaged 2.11 points. The Bucs were identical, with 2.11 points per drive.

I think “missing plays” is what EPA captures better than anything. When a team doesn’t score, for whatever reason, EPA captures and penalizes the offense, whether it’s a fumble, an ugly pick, an “arm punt,” or a normal punt. It’s agnostic on the actual method of failure, it just quantifies the failure.

Many of these concepts were introduced in The Hidden Game of Football by Bob Carroll, Pete Palmer, and John Thorn, which was published way back in 1988. They didn’t have the fancy tools we have now to make this easy, but what they were after is in the title of the book. What’s missing is almost always as important as what’s there, and while we’re trained to notice Jameis Winston’s picks from years of exposure to the basic NFL stats, we’re not as trained to notice Aaron Rodgers checking down or throwing the ball out of bounds on third down. Those plays have a cost, and while most assume it’s less than a pick, they add up.

If you are curious about playing around with EPA, it is included in Pro Football Reference game summaries (just beware of Expected Points After v. Expected Points Added), but there are also some nifty, free tools that make this much easier than even 12 months ago. Michael Chiang, now of The Athletic as well, has an open-source database with a very user-friendly interface that anyone can use:

There is also airyards.com, which is relatively simple to use and is invaluable for quarterback analysis including CPOE, and anything pertaining to depth of target. And, if you want to do your own deep dive, you can always download R, for free, and get NFLScrapR, which I did for my two pieces on timeout usage last season.

Advanced stats made huge strides in 2019, and most NFL teams now have statistically-minded people at some level in the front office. As in baseball, they will probably be running things in short order. If you want to stay ahead of the pack as an analyst, knowing EPA and its brethren will shortly be an absolute must.