The S-2 Cognition Test and You

One of the big winners of this pre-draft cycle is undoubtedly the S-2 Cognition test, which is now frequently cited to boost (or in one high-profile case, to denigrate) the quarterback prospects in this class. I suspect we will be hearing about the S-2 for quite some time, and so it’s important to understand what it is, what it is not, and whether we can trust it.

What is it?

If you’re interested in the S-2, I recommend checking out this piece by Matt Barrows in The Athletic,, this piece by Dalton Miller at Pro Football Network, and this podcast featuring one of the founders of S-2. The S-2 is a battery of tests intended to gauge how quickly and accurately a prospect processes information in a chaotic environment while ignoring distractions. In the words of S2:

S2 Cognition attempts to quantify split-second reaction times to visual stimuli on a computer screen. The cognitive skills measured by the S2 evaluation are the same cognitive skills engaged on the field, court, or ice. Our evaluation is devoid of racial bias, it does not measure IQ, intelligence or classroom type knowledge, and it cannot be studied for. It requires simple reactions to quick moving visual stimuli on the screen.

What it is not

First and foremost, the S-2 is not the Wonderlic. The test first came to my attention the aforementioned article in The Athletic (which is well worth a read), and the reaction online to those who just saw the headline was instantly to draw a comparison to the the Wonderlic, which is more like a standard IQ test. S-2 has noticed this comparison and goes out of its way on their website to distance themselves from the Wonderlic and similar tests.

This isn’t just lip service either. The Wonderlic is a timed intelligence test, and it’s never meant anything in terms of projecting football players, though it has been used to embarrass them. The S-2 is more like a video game than an intelligence test, and is, in fact administered via Xbox:

“The S2 Eval requires taking the test on a specialized laptop or XBOX.”

The test itself involves quickly identifying shapes and patterns and at no point is anyone answering questions. It’s about vision, reaction time, and spatial awareness, and in that sense, is closer to a physical drill than a test.

That hasn’t stopped people on the internet from using the results to denigrate certain players, similar to what has traditionally happened with the Wonderlic, and more than anything, it hasn’t stopped Bob McGinn, who has made questionable use of the Wonderlic in the past to impugn various prospects, and has gleefully jumped in with the S-2.

Can We Trust It?

The short answer is, absolutely not, for a variety of reasons, but before we get into the substantial negatives, lets deal with the positives. The Wonderlic was never intended to measure any specific football skill or trait. S-2, at least, seems crafted to detect skills that a quarterback should have. Quarterbacks are tasked with making lightning fast decisions by processing multiple layers in their field of vision, all while large men in their peripheral sight are attempting to kill them. The speed at which they can do so seems very important, and also, I would argue, seems like something that could be measured.

I think S-2 has a good idea here, and the seems to agree as they are, presumably the ones paying for it. That doesn’t necessarily mean it’s valuable – they paid for Wonderlic too – but it’s not nothing.

They also count a number of great NFL quarterbacks among their successes including Drew Brees, Patrick Mahomes, and Josh Allen. Allen is particularly intriguing as the analytics world almost unanimously predicted failure for him, while his S-2 results got it right. The founders sound impressive in discussing their product, the concept makes some sense, and there is some track record of predictive success.

So what’s the problem?

S-2 can tout their successes all they want, but their proprietary nature means we don’t have a full set of data to actually verify the significance of their results. There are plenty of pseudoscience-y tests that sound good in theory, from Wonderlic, to Myers-Briggs, that will look great when only presented with the successful sample. We’re not the clients, and we’re not entitled to that data, but without transparency, they’re not entitled to trust either.

I have some additional problems as well. While S-2 touts plenty of obvious success stories, we should not gloss over Brock Purdy, who is the focus of this article in the Athletic, and Chicago Bears’ quarterback Justin Fields. Purdy was a brief sensation last year before his elbow was destroyed, but success in the Shanahan 49er system is hardly definitive. The mediocre Jimmy Garoppolo has been incredibly efficient in that offense for years, and even the atrocious Nick Mullens has been semi-competent. It’s hardly a certainty that Purdy is any good, and we should not count our chickens before their elbows are surgically reconstructed.

Fields is, perhaps, more damning due to his specific problem. This test is, generally speaking, supposed to detect quick decision-making, and it’s hard to imagine a worse case for it than Fields, who has struggled with making quick throws even when playing with elite talent at Ohio State. If we are to believe the leaked S-2 score for CJ Stroud, it’s hard to square his style of play, and test scores, with what we know of Fields.

I have a few additional issues. In The Athletic, Barrows wrote the following about Bryce Young:

I do have a feeling that a quarterback from Alabama that we have tested every year since he was in 10th grade may end up sharing his results publicly because he actually owns those results and the NFL does not,” Ally said.

While S-2 states that you cannot practice for their test, that’s obviously an enormous oversimplification. While they may believe you cannot train certain skills, (and they clearly do believe you can train some of these based on literature on their website), familiarity with the tests and how they work will give you an obvious leg up. Bryce Young having done this test so often will give him an enormous advantage over someone taking it for the first time. Even if the substance of the test is random, knowing how the controls operate, what the various figures look like, how the game starts and stops, and what the distractions and noise are, will be helpful. Someone taking the test for the first time has to figure out the game in addition to performing well.

On the Zierlein podcast, they also make a big deal out of just how granular their measurements can get, and how significant the difference is between a few microseconds can be. Maybe they’re right, but I’m always skeptical of extremely small time measurements. In high level swimming, they specifically do not measure to the thousandths place because differences in thickness of the paint on the side of the pool can have an impact at that level. The smaller the time frame, the more outside factors can impact the timing.

And finally, we don’t even know what we can and cannot trust. Maybe McGinn was lied to, but we have conflicting reports on Stroud.

Without official word from the NFL or S-2, or the quarterbacks themselves, we not only lack information, we are also incorporating actively bad information generated by the pre-draft media manipulation cycle.

There are simply too many unknowns, and too much noise to put too much stock in this. We don’t know who has taken part it in the past, and who hasn’t, and how everyone fared. We don’t know of counterexamples that may have performed poorly, but succeeded in the NFL. They use passer rating frequently on the S-2 website, which is not what I would look to for a success metric in the first place.

Finally, Eric Eager made this excellent point on the subject on Friday.

Young and Stroud have been of roughly equal value the last two seasons, even when adjusting for the disparity in surrounding talent, and strength of schedule. If this test really matters, Stroud should not have been as good in college. But he was.

The S-2 may be an interesting data point, and I’ll certainly be keeping an eye on Jake Haener’s career as he scored extremely well. But it is a small data point. Until we know more, we should take it with many large grains of salt.