Quarterback Scarcity Is Over, Part 3: Analytics struggled with college QBs, but it’s getting better

Read Part One and Part Two of this series here.

---

Analytics broke in through baseball before invading every other sport, but the concept has taken a very different path in football. While the actual questions being answered were largely the same — those questions being what statistics are important to winning and which players possess traits to drive those statistics — analytics entered baseball front offices primarily as a method of player acquisition. Baseball teams shied away from the many obvious on-field strategic uses of analytics for a long time, and while it’s now common to discuss shifts and the “times through the order penalty” during the course of a game, it took a while to get there.

Because analytics were more or less created through baseball, that sport experienced a time when they were relatively primitive, limited by available information and technology. The simple discovery by Billy Beane’s Oakland Athletics, as detailed in Michael Lewis’ book Moneyball, that on-base percentage was an undervalued statistic, is a pretty simple thing. What can be measured, quantified, and evaluated by smart people with big databases has grown by leaps and bounds over the years, and as baseball has moved on from teaching catchers to steal strikes through framing to changing the way swings are taught, every other sport now benefits from skipping those dark ages and gets to jump right into the good stuff.

In the NFL, analytics stormed in on all fronts, quickly making their presence felt on the field (the superiority of passing v. running, fourth down aggressiveness, the relative success of certain types of plays like play action passing), in the front office (RAS, Sparq, Ted Thompson’s Thresholds, etc.) and in player development.

All of that said, cracking the quarterback code has remained tricky business. While we have a pretty good understanding the quality of NFL quarterbacks, projecting college quarterbacks has typically been difficult. Every time analytics seems to have figured something out about college prospects, college football changes just enough to make it more complicated, and that is where our story begins.

Let’s talk about Brian Brohm.

Brian Brohm, Aaron Rodgers, and Matt Ryan

In the offseason of 2008, Aaron Rodgers, at the time a 3-year veteran backup after being selected in the 2005 draft, was no sure thing to take over for the Green Bay Packers. In the 2008 draft (in which the Packers picked up Jordy Nelson, Jermichael Finley, and Josh Sitton), Green Bay used a 2nd round pick on Louisville’s Brian Brohm. Brohm was a controversial prospect, but more than anything, analytics people loved him.

David Lewin of Football Outsiders spent some time attempting to figure out what the most important college statistics were for projecting quarterbacks in the NFL. Lewin landed on the following:

1. The QB needed to have started at least 33 college games
2. Have a completion percentage of at least 58%, with higher being better, and
3. Be drafted in the second round or higher.

Number three served to limit the sample to only quarterbacks adequately vetted by scouts, and it makes some intuitive sense that more experienced quarterbacks with high completion percentages would be successful. Lewin’s system did a pretty good job “projecting” success from the sample of quarterbacks from 1997-2005, but then things changed, and the major culprit was Brohm.

For the statistically minded among you, I’m sure you are already picking Lewin’s criteria to death. You’re correct to do so, and many of the inherent problems here (selection bias, a statistical cutoff for a stat that has been increasing since the invention of the forward pass, more selection bias, etc.) are exactly why the stat ultimately failed. It was a different time.

Lewin’s system loved Brohm, and even among many scouts he was seen as a potential first rounder, but that praise was far from universal. Brohm was a big test for the system, and when he fell to the second round, the Packers snapped him up. They would add Matt Flynn in the 7th, and we all know what happened next. Rodgers became an all-time great, Flynn became a solid backup, and Brohm busted out of the league.

In terms of college statistics, Brohm blows Rodgers out of the water. He completed a higher percentage of passes (65.8% v. 63.8%) at a higher yards per attempt (9.1 v. 8.2), and Brohm was a 3-year Division-1A starter compared to Rodgers’ 2 years plus a junior college season. When it came down to their professional careers, why didn’t their college production matter? Why did Brohm’s elite college accuracy not translate to NFL success, and has the analytics community improved itself since 2008?

Getting Granular

While this article is primarily about analytics, I would urge you to take a moment to watch some college highlights from both Brohm and Rodgers. I’m not asking you to scout off of highlight packages, as they are by their nature designed to make each player look good. What I want you to pay attention to is the complete contrast in style between the Louisville offense and the Cal offense, and between Rodgers and Brohm. In many ways, they’re barely playing the same position. Brohm plays almost exclusively from the pocket and benefits from a ton of pre-snap motion and play-action. His highlight reel is littered with long, looping bombs. Rodgers, on the other hand, is constantly rolling out, buying time with his mobility and firing darts all over the field, while holding the ball absurdly high for some reason. Those stylistic differences are important.

Lewin’s system takes a broad view of quarterbacks, expecting that completion percentage and experience will mean the same thing to any passer over at least 33 games. It may very well be the case that “accuracy” is one of, if not the most important trait in a quarterback, as captured in Lewin’s system by completion percentage, and in more modern settings as Completion Percentage Over Expected (CPOE), but it is just as important, and perhaps more important, to answer the question of why a college quarterback was accurate or inaccurate. The context around Brohm and Rodgers could not be more different, and that is ultimately why Lewin’s system and many subsequent systems failed. At a macro level accuracy is important, but in an age when college talent is extremely imbalanced and college coaches are adept at making things easy on their quarterbacks, macro statistics struggle to capture true accuracy.

If analytics is to solve this puzzle, it would require the granularity of a scout.

Why was Brohm accurate in the first place?

Let’s start by breaking down just who is responsible for a quarterback’s production in the first place. As it happens, Brohm had a lot working in his favor back at Louisville

1. Scheme

Brohm is a pure pocket passer, almost never rolling out or scrambling. On his big plays, he’s almost always making quick reads and getting the ball out on time. Louisville was also running an extremely modern offense under Bobby Petrino, even by today’s standards, making frequent use of pre-snap motion, play action, and all of the fun stuff that analytics people love. No high-level college quarterback had it easier from a scheme perspective than Brohm, and he was really only called on to make a few simple reads and deliver an accurate pass to a predetermined location, often with little pressure.

Brohm wasn’t a bad athlete in terms of combine metrics (he actually had a better RAS than Rodgers), but he didn’t make use of any mobility he did possess on the field, and scouting reports of the day routinely characterize Brohm as lacking arm strength.

Arm strength isn’t everything, but when a quarterback is succeeding in spite of a poor arm, it’s often a signal that he has a schematic advantage. One of the better ways to compensate for a slower ball is to know where the ball will be going earlier than you normally would. Sometimes a quarterback can compensate with a quick release, or elite processing through progressions. Watching Brohm, it’s clear that he knows where to go before the ball is snapped on most occasions. Since he’s not making reads during the play, the scheme is the big factor.

The nail in Brohm’s prospect coffin was the simple fact that we got to see him play his final season without Bobby Petrino, as something called “Steve Kragthorpe” took over the program in 2007, taking the Cardinals from a 12-1 juggernaut to a 6-6 also-ran. Kragthorpe coached the Cardinals until 2009, getting progressively worse each year. As a senior, Brohm’s Y/A dropped by over a yard, and while he did hit a career high in attempts, his interception total also more than doubled from 5 to 12. Brohm was much less efficient without Petrino, which raises the question of just how much of their earlier success was actually due to Brohm.

Finally, on top of everything else, we should briefly mention Stefan LeFors. You may not know who Stefan LeFors is, but if you’re a college football fan, you probably should. Quarterbacking has gotten much more efficient over the past 10 seasons with new training techniques, schematic developments, and a focus on high completion percentages and low interceptions. If you export a list of all 923 college quarterbacks with at least 400 attempts in their career from the year 2000-present, Lefors’ career efficiency rating ranks 38th. This is especially impressive given that almost every quarterback higher than him, with very few exceptions like Sam Bradford and Tim Tebow, played in the last 5 years.

In 2004, LeFors had one of the single greatest seasons a college quarterback has ever had, completing 73.5% of his passes at 10.1 yards per attempt, with 20 TDs and only 3 interceptions. Despite this phenomenal senior season, Lefors was a non-prospect, and while he was drafted in the 4th round of the 2005 draft by the Panthers, he never caught on, mostly bouncing around the league before moving to the CFL.

LeFors was the quarterback at Louisville before Brohm took over. LeFors ranks higher in career efficiency rating, and Brohm never had a season as good as LeFors’ 2004. LeFors was great in all the same ways that Brohm was great. Which is to say, Bobby Petrino was great.

2. Talent

Louisville wasn’t all Bobby Petrino, as Brohm also had the luxury of targeting two future NFL players in Harry Douglas, a solid contributor for the Falcons and Titans for a decade, as well as tight end Gary Barnidge, who had one of the best NFL tight end seasons of all time in 2015, and was otherwise fine.

For Aaron Rodgers, his supporting cast was “college good,” but the NFL talent at Cal was tied up at running back in the form of Marshawn Lynch, and JJ Arrington. That’s a fantastic duo to have for the running game, but Rodgers’ best receiver, by a country mile, was Geoff McArthur, who never played in the NFL. McArthur dominated targets over Rodgers’ two years, as there was no true secondary option out wide, and much of the responsibility of taking pressure off of McArthur fell to Arrington. Brohm certainly had better targets to work with.

It’s also worth taking a look at Matt Ryan, who was taken 3rd overall in the 2008 draft along with Brohm and Joe Flacco, and turned into a pretty good NFL quarterback. Ryan had generally terrible numbers as a senior (under 60% completion percentage, under 7 yards per attempt). Scouting off of statistics alone, Brohm was clearly superior, but again, we have to look at Ryan’s situation. Ryan played his first three seasons under Tom O’Brien before he was replaced by Jeff Jagodzinski. Jagodzinski’s pass-heavy offense was a success in terms of offensive output, but it didn’t do Ryan any favors statistically as Boston College’s personnel was built for a short passing attack (in fact, running back Andre Callendar led the team in targets) while Jagz preferred the deep game. Ryan’s completion percentage suffered at the mismatch of style and personnel, but fortunately for Ryan, scouts didn’t care, and they won the day. Again, Brohm’s situation was much better.

Can analytics solve this?

By virtue of a great historical accident involving FoxTrax, the technology that put a glow around the hockey puck while broadcast on Fox, we have unprecedented access to statistical information about baseball. That puck tracking technology eventually became Pitch/FX, and at first, MLB didn’t realize quite how valuable that data would be. Once Pitch/FX data became publicly available, it became difficult to take it back.

In football we have plenty of good data freely available via various analytics websites like Football Outsiders, SIS, PFF, Next Gen Stats, or via NFLScrapr, but the good stuff is tougher to get publicly. Player tracking data exists and is parsed out in small chunks via NextGenStats, while SIS and PFF offer charting data in their premium packages. But behind the scenes, teams have an unprecedented amount of data and know how to use it.

There are two keys to figuring exactly what you should be looking at when projecting college quarterbacks to the pro game. First, you have to figure out which throws/skills translate to the NFL, and then you have to figure out whether or not that skill is actually important. At this point, someone is tracking and compiling data on every type of throw a quarterback can make, how often the quarterback throws on the move, from the pocket, from shotgun, under pressure, and dozens of other variables.

In their recent rookie handbook previewing the quarterbacks available in the 2021 draft, Sports Info Solutions’ John Shirley breaks down every prospect using catchable% (how often the quarterback delivers a catchable ball) and on-target% (essentially tracking “good” throws where the receiver doesn’t need to make any substantial adjustment to haul in the pass.) Each metric attempts to separate the quarterback’s performance from that of his supporting cast, which is especially important with the talent disparity between schools like Alabama and Ohio State versus Trey Lance’s North Dakota State.

They don’t stop there though, as catchable% and on-target% don’t adjust for the difficulty of the throw being made. Setting a baseline for comparison is one of the most important aspects of analytics. Most advanced baseball statistics use “Replacement” as their baseline (as in Wins Above Replacement, or WAR). Replacement value is the hypothetical value that a freely available player would provide if added to a team right now. Football Outsiders DVOA statistic uses “average” as the baseline. Recently, a new contender has entered the field in the form of “Expected Outcome.”

CPOE is probably the most famous football statistic based on expected outcome, and it’s a huge step up from normal completion percentage. Essentially, the statistic uses a model (based on years and years of earlier NFL data) to determine how often a given pass should be completed based on depth of target, quality of competition and various other factors. We can then compare the player’s actual completion percentage to what they “should have” completed.

You can also find “expected-based” statistics on the Next Gen Stats page for running backs, where Nick Chubb, JK Dobbins, and Aaron Jones dominate. SIS has their own proprietary version of expected outcomes called Predicted Completion Percentage (Pcomp) and Expected On-Target Percentage (xOnTarget). They then compare their baselines to actual completion percentage and on-target percentage to generate pComp+/-, and xOnTarget+/-. I’ve chosen to go with SIS here because, unlike CPOE, their data is publicly available in their annual, where they detail the upcoming quarterback class. SIS is hardly the only option as PFF publishes similar data, and it’s useful to have a few different methodologies both as competition, and as a check on accuracy. SIS is quick to point out that pComp has a 0.9 correlation with CPOE.

The 2021 QBs

Here, we get some interesting results. Justin Fields ranks highly in pComp+/-, but struggled in making on target throws compared to his peers. Per SIS, Fields had a ton of help from his receivers compared to the other major prospects, as they were able to haul in 23% of his off-target throws. On the other hand, Kyle Trask, who generally ranks lower as a prospect in this class, ranks very highly in OnTarget%. One of the strangest results from SIS tracking is the relative dominance of Feleipe Franks, whom Trask replaced at Florida and who seems to score well regardless of what metric you select. Good projection systems should generally tell you what you already know, plus a few surprises. If Franks ends up being something, it will be a feather in their cap.

SIS has also done some work identifying which traits translate from college to the pros at all skill positions; these include throws made under pressure, throws from shotgun, throws on non-play-action plays, and shorter throws under 15 yards. The correlations aren’t that strong, as football is a noisy game, but they are there. That said, this level of granularity is somewhat new, and at least with SIS, we don’t know how well these are going to work going forward. As we discussed in Part 2, quarterback development is vastly different now from what it was even a few years ago, and what could not be fixed before may be fixable now. Effectiveness under pressure shows a strong (for football) correlation from college to the NFL, and short passing effectiveness shows a similarly high correlation. So what to make of Justin Herbert?

Justin Herbert as Anti-Brohm

Even if some of these things correlate, that doesn’t necessarily mean that they are unchangeable, Let’s take Justin Herbert, formerly of Oregon and now of the Los Angeles Chargers, as an example. In college, Herbert was one of the single worst quarterbacks under pressure. Way back in 2020, Pro Football Focus graded Herbert as the 5th-worst college quarterback under pressure, writing:

“Though if the clean pocket collapses, Herbert becomes increasingly volatile. Among 129 qualifying quarterbacks this season, Herbert ranked 124th in negatively graded play rate under pressure. You can see his panic in a collapsing pocket, an area where a first-round quarterback really shouldn’t be losing his poise. He’ll try to create outside of the structure but will toss up some desperation heaves and, in turn, produced the 47th best accurate-pass rate among 66 qualifying quarterbacks.”

And Herbert struggled on shorter throws in college.

“Quick, short passes have been Herbert’s nemesis this season, ranking 64th in the FBS in accurate-pass rate on throws in 2.5 seconds or less and traveling just 1-9 yards. On throws of 19 yards or fewer when clean, Herbert has thrown the fourth-most incompletions that were faulted to him — whether it was an overthrow, underthrow, ball in front or behind, etc. Regardless of the scenario, he owns the second-highest rate of such throws among the top seven quarterbacks in the 2020 draft class.”

These are subjective observations of PFF’s charting team, however they were hardly alone in noting these issues. One of his big statistical faults was simply that he wasn’t as productive as similar quarterbacks with similar accuracy numbers. Yet, his excellent rookie season tells a different story. Again, from Michael Renner of PFF:

“We’ve never seen a better performance from a rookie under pressure and, honestly, it’s not even close. His 75.4 passing grade under pressure was not only the highest we’ve ever seen from a rookie, it was also the highest in the entire NFL this season. Of the 30 seasons we’ve seen where a quarterback earned a 70.0-plus passing grade under pressure, only five of them came from players you wouldn’t immediately recognize as sure-fire, stud, franchise quarterbacks: Jeff Garcia (2007), Robert Griffin III (2012), Carson Palmer (2015), Jay Cutler (2015) and Marcus Mariota (2017).”

Both Renner, and SIS author Alex Vigderman, who penned the article in the SIS Rookie Handbook on statistical correlations from college to the NFL, are quick to point out that effectiveness under pressure in the NFL itself is actually highly volatile from year to year. If you were looking for a reason that Herbert might regress in 2021, his poise under pressure is a good place to start. That said, we can’t sneeze at a quarterback going from one of the worst, to one of the best in a single season.

There’s one aspect of analytics that makes this all the more difficult. Statisticians can analyze what a player does well now and how they will likely fare later, but in identifying a players’ specific deficiencies, they can also pinpoint areas for development. If players are improving on their specific deficiencies because of NFL coaching, those correlations will disappear pretty quickly and that may be what happened here. Herbert is similar to Josh Allen in that he has a huge arm and he’s quite mobile. It also looks like the scheme at Oregon held him back a bit. But it’s also possible that Herbert wasn’t actually that good as a rookie.

Herbert occupies an interesting place in the upper left quadrant of this graph, populated mostly by past-their-prime old men. Being in the upper left means that a quarterback produced positive value on a per-play basis despite completing fewer passes than expected. Drew Brees lives in this quadrant because he was heavily reliant on scheme and YAC. You might think, given Herbert’s physical status as a big guy with a strong arm, that Matthew Stafford might be the best comp for the rookie, but if we take a peek at last year’s Air Yards results, it’s almost certainly the aged (and now-retired) Philip Rivers.

That’s not great news for Herbert, as Rivers thrived by playing in an outstanding scheme developed by Frank Reich, and checking down a lot. It is a positive that Herbert’s short accuracy has improved, and putting up a positive EPA is a good thing regardless of how you get there. For a rookie, it was a nice season no matter how you slice it, and his apparent development is commendable, but we should be cautious.

Quarterback Scarcity is Over

With all of that said, I wonder how wrong the analysts actually were on Justin Herbert and Josh Allen. You can measure so many traits and slice and dice so many metrics for each player, that it’s virtually impossible to have a poor scouting report on anything other than makeup. Analysts can point, over and over, to how often certain player types have failed, but teams aren’t static. Most teams, at least the sophisticated ones, likely have a good idea of exactly what they can and cannot develop. The recent focus on mobile quarterbacks reflects a larger trend in the NFL that shows more faith in a team’s ability to develop a prospect. College football has featured great athletes forever, even (especially) back into the triple option days, with the only difference that a college team never required as much development.

Patrick Mahomes is the NFL’s best quarterback,

and Josh Allen may be his equal in short order.

Herbert had a nice completion percentage in college, but he also has a cannon, and an outstanding RAS:

Once you diagnose deficiencies like poise under pressure or short throw accuracy, you can use your coaching staff to specifically work on those issues. You can make additional use of that strong arm and mobility to scheme easier throws. In short, analytics, more than projecting fully developed players, allows you to more accurately assess a players’ strengths and weaknesses and develop a training program and scheme to maximize their gifts.

Quarterbacks are better than they have ever been before, but they’re also more diverse. Nowhere is that more apparent than in this class, where Alabama’s Mac Jones stands as one of the most accurate quarterbacks in college football history, but also one of the least mobile and least physically gifted. Justin Fields is an incredible athlete, but he was helped by his surrounding talent, per SIS, more than just about anyone else. You have your “can’t miss” guy in Trevor Lawrence, and your scout’s dream in the small school prospect Trey Lance. Using one single criteria to project this field is a fool’s errand, but thanks to an unprecedented effort in data collection, no team should be surprised by what they are getting.