Did Matt LaFleur really make the worst coaching decision in Week 10?

One of the weirdest things about the Green Bay Packers’ game against the Jacksonville Jaguars in week 12 was an analytical evaluation of a alte-game decision by head coach Matt LaFleur. EdjSports, which among other advanced statistics runs win percentage calculations for coaching decisions every week, decided that Matt LaFLeur made the worst coaching decision of the week in that game:

Often, the worst coaching decision of the week is pretty obvious. There are plenty of bad NFL coaches and they often make bad decisions. This one doesn’t really stick out. EdjSports says that the Packers’ final punt of the game, on 4th and 1 from their own 23 yard line while leading 24-20 with 2:32 left in regulation, was the worst decision of the day. Not only does that decision not seem particularly dumb at first glance, it seems...correct? Edjsports claims that punting rather than going cost the Packers about 18 points of win probability. That seems like a lot.

Surrender Index didn’t even take note of that punt, and identified this Jacksonville punt as the most cowardly of the game:

And data from Ben Baldwin’s RBSDM site sees only a 1.5 point change in EPA from the punt, and a 3.3% change in WP. Now, Baldwin’s calculator doesn’t give us the expected outcome of going for it, which is the rub, and why it’s worth discussing. ProFootballReference.com (PRef) does offer up a win probability calculator, which we can at least use as a reference point for the EdjSports. Per PRef the Packers’ win probabilities are:

• Punt: 77.62
• Go and succeed: 95.57
• Go and fail: 66.66

Given these numbers, Edjsports’ criticism doesn’t seem that out of line. The Packers aren’t great on 3rd and 4th and 1, but the upside of making it is pretty substantial.

It’s not 18 points higher though. It would be if the Packers were guaranteed to make it, but given the possibility of a worse outcome about 40% of the time, the true gain is less than that, at least according to this model.

The EdjSports description is also ambiguous as to how they arrived at their number, and so I contacted Frank Frigo of EdjSports for some additional clarity. Frank writes the accompanying “Risky Business” column for EdjSports explaining their analysis, and he was more than happy to walk through their methodology in some detail. EdjSports uses Monte Carlo simulations for every game-state that occurs over the course of a game to produce win probability estimates going forward. Using these simulations, they can compare the optimal strategic outcome of their projection to the actual decision made by coaches during the game. They have the ability to factor in fairly specific things, like Jake Luton being a terrible quarterback or the environment not being conducive to passing, and it is this additional context that creates differences between their projections and a more static calculator like the one at Pro Football Reference.

The actual article criticizing LaFleur’s decision seemed to reference the outcome of JK Scott’s terrible 30-yard punt, but Frank confirmed that is not the case and that all of their analysis is based on the model’s forecasts prior to the play. Scott’s actual punt was worse than they projected, and so the actual outcome cost them even more in terms of winning percentage in real life than they projected.

Frank was also quick to point out some potential scenarios that I hadn’t considered. For instance, had the Packers failed to convert and the Jaguars managed to score the go-ahead touchdown, there likely would have been time for Aaron Rodgers to lead a comeback needing only a field goal to tie. He also pointed out that while it may seem like facing a poor quarterback like Luton may support more conservative decision-making intuitively, that isn’t necessarily the case, and that expected poor outcomes from the Jacksonville offense can actually support a riskier offensive philosophy, as they did here.

Think of it this way. Pro Football Reference knows that a two touchdown underdog has a 33% chance to win if the Packers go for it and fail. The reality was that Jacksonville was likely worse than PRef is capable of understanding on offense. Luton had virtually no success on the day, and the EdjSports model is privy to that performance to make its projections. That cuts both ways of course, and they were probably also less likely to score after a punt (and in fact, did not score), but intuitions on preferring a risk-averse strategy are not really supported.

According to Frank, they’ve rigorously tested their model against real results, and at least part of their business model involves selling this information to bettors. I’m generally in favor of going for it on 4th and short almost anywhere on the field and so I don’t find it that surprising that an analytics-focused website would call this, or any similar decision, bad. That 18% number seems pretty big at first glance, but given the baseline we created with PRef and the additional game context they can incorporate, it does make a fair amount of sense. As the game inches closer to concluding, every play’s impact is magnified. In a close game, it’s not uncommon for WP to swing wildly from play to play, and every decision is more important. A coach can come back from a bad decision in the first quarter, but late in the fourth, it is more difficult to do so. As a result, a risky but-correct decision can become more correct as the game winds down.

So, I think EdjSports has point, but I do have to mention one thing Frank told me that’s still bugging me. He said that according to their model, the Packers should have gone for it even if their chance of succeeding was as low as 15%. He said that they ran model simulations at both extremes, and that even in a scenario where the Packers could have counted on getting a massive JK Scott punt that pinned the Jaguars at the 1 yard line, that it still would have been correct to go for it if they would be 56% likely to convert (which is, coincidentally, their approximate short-yardage conversion rate). Possession is obviously important, but this strikes me as a bridge too far.

For one thing, EPA at your own 1 yard line is negative. You are more likely to be scored on than to score yourself, and such a punt would put the Jaguars into this scenario with 2 timeouts and the two minute warning left. PRef’s calculator gives the Packers an 88.68% chance to win in this scenario. That isn’t 95.57%, but is pretty close, and seemingly would require more than a 50-50 shot to make going for it preferable. It’s essentially a moot point as no such punt is coming off the foot of JK Scott, but it seems like they are protesting a bit too much.

EdjSports has seemingly put the work in, and more than anything, I think we need to consider the reality that on 4th and short, it’s almost always worth going for it. Extraordinary claims require extraordinary evidence, but after looking more closely at other win probability calculators and EPA models, they have a point.