Some viewers of Amazon Prime’s “Thursday Night Football” matchup between the Kansas City Chiefs and Denver Broncos got a special experience: seeing what was going to happen on a play before it actually happened.
Viewers who watched the game in Prime Vision with Next Gen Stats, one of Amazon’s three broadcast options, saw the unveiling of a feature called Defensive Alert that is powered by artificial intelligence to identify potential blitzes before the snap. The model highlights players it believes have a high probability of blitzing (crossing the line of scrimmage to rush the passer) with a red circle that appears under them.
Advertisement
As Sam Schwartzstein, one of the minds behind the model with Prime Vision and the Amazon Machine Learning team, eagerly watched his model work its magic on every snap, on one play, he became confused at why his program was highlighting a nickel corner who wasn’t giving any indication that he was blitzing. “Why are we highlighting this guy?” Schwartzstein yelled in fustration.
![go-deeper](https://cdn.theathletic.com/cdn-cgi/image/width=128,height=128,fit=cover,format=auto/app/uploads/2023/10/17112143/GettyImages-1725117157-1024x683.jpg)
GO DEEPER
The Eagles' 'tush push' play is taking over the NFL. But is it working?
Was this a flaw? Did the machine get this prediction completely wrong? Schwartzstein, a former offensive lineman who played at Stanford with Andrew Luck, prided himself on being able to identify potential blitzes. His years of experience as a player and analyst told him the nickel wasn’t much of a threat.
Right before the ball was snapped, the inside linebacker dropped to the nickel’s side and the nickel finally moved toward the line of scrimmage. The program sniffed out this blitzer well before Schwartzstein, watching the game from a wide-angle camera shot. The weirdest part about this is no one really knows how Defensive Alert did it. It’s a self-learning program that has analyzed thousands of plays and movement patterns to understand how defenses move as a whole when certain players blitz.
ALERT! ALERT! 🚨
The new Defensive Alert feature available exclusively on @TNFPrimeVision w/@NextGenStats identified Patrick Surtain II as a defender to watch off the edge before the snap.
📺: https://t.co/1OMRlplD2C pic.twitter.com/gzPe21ilNH
— Next Gen Stats (@NextGenStats) October 13, 2023
The model is trained not to identify the usual four down linemen that typically rush the passer. It’s trained to identify unique players who rush the passer on 60 percent or less of snaps. It’s being fed tracking data from Next Gen Stats, which is derived from RFID chips in every player’s shoulder pads. The data includes the players’ acceleration, their orientation and where they are facing. From all that data, the machine starts to understand familiar movement patterns from the defense as a whole, which helps it predict which player is going to blitz.
Advertisement
“We’re highlighting things, starting at line set,” Schwartzstein said. “It’s happening in real time as information is coming in from the shoulder pads. And so you can see all this data coming in and (the model) gets more confident the closer we get to the snap because defenders have to more clearly define their roles the closer they are to the timing of the snap. One of the coolest features for me is we’re not just highlighting it at one time and sticking with it. It is on and off based on where players are moving throughout the play on both offense and defense.”
Defensive Alert, along with many other features on Prime Vision, has turned “Thursday Night Football” into a living, breathing broadcast that is helping the audience see all the nuances of the game as they are happening live.
The goal is to get viewers to see the game as the quarterback does. The quarterback isn’t certain who is going to blitz, especially early on when the defense is showing its initial disguise. But as the snap nears, players start to move around to get close to where they have to, to execute their assignments. The initial alignment and movement help the quarterback figure out who is blitzing or not. The best quarterbacks are coming to conclusions from their wealth of experience or film watching. The machine is processing information the same way, but it has an abundance of data that has been fed into it to pull from in an instant.
Some skeptics believe Amazon is using a delay to see the blitzes coming and highlighting the player on the live feed. Though the processing required for Prime Vision to paint visuals does add some delay (usually three seconds or less), the model that powers Defensive Alert does not use that delay. The team has spent considerable effort to produce predictions as fast as possible — even installing dedicated hardware in Amazon’s state-of-the-art production trucks. There is no person or program trying to trick the audience about prediction capabilities.
One of Schwartzstein’s favorite moments in the game came when the Broncos brought a Cover 0 (man-to-man with no deep help) blitz, meaning they were bringing the house, late in the game.
![](https://cdn.theathletic.com/app/uploads/2023/10/17154321/Screenshot-2023-10-17-at-12.39.16-PM.png)
The Broncos came out with a pressure front with both linebackers mugging the line of scrimmage and a safety showing blitz. The model highlighted all three players, as they were obvious blitz threats. But it also highlighted the nickel corner (top of the screen) even though he was lined up over the slot receiver. The model likely saw the safety lined up directly behind him and recognized this as a nickel blitz indicator.
At this point, you can hear Patrick Mahomes scream out, “Watch Cover 0! Watch Cover 0!” on the broadcast. Before he did, the audience could already feel and see the potential blitzes with all of the players highlighted.
After the ball was snapped, the right tackle blocked the wrong defender, allowing a free runner to get to Mahomes. One of the inside linebackers whom the model highlighted dropped, but the other three highlighted players blitzed.
Correctly identifying the linebacker who dropped from a mugged position would have been extremely difficult, but there might be a time when the model can do this as it keeps getting fed more data.
That play made it relatively easy to identify potential blitzers. Let’s look at the play in which the model saw a blitzer coming before Schwartzstein did.
(The names of the defensive players weren’t shown on the broadcast. I added them in to make it easier to follow along.)
Here, the defense had inside linebacker Nick Bolton mug the A-gap. Nickelback Trent McDuffie lined up outside leverage of the slot and was 4 yards away from the line of scrimmage. Right as the offense lined up, the model already identified McDuffie as a potential blitzer. Schwartzstein didn’t know why. He said before Bolton dropped, he would have kept the protection inside if he were on the Broncos offensive line rather than push out toward McDuffie.
(The arrows showing the defensive players’ paths weren’t shown on the broadcast.)
Right before the snap, Bolton dropped toward the three-receiver side and McDuffie advanced toward the line of scrimmage. At this point, it would have been too late to change the protection. There are protection rules that can ultimately lead the offensive line to pick it up, but it’s always advantageous to have a plan to pick up blitzers before the snap.
Both nickels blitzed, and the Broncos offense didn’t see it coming either because McDuffie blew right by the tackle, who was focused on an inside rusher.
After this play, Schwartzstein texted his producer, “I just lost to the model.”
Again, Schwartzstein doesn’t know exactly how the model is making some of these predictions. It’s learning on its own as it keeps getting data, but don’t worry, football purists. The model is also getting input from a panel of actual football people that includes former players and coaches like Andrew Luck, Geoff Schwartz, David Shaw, David DeCastro, Ryan Fitzpatrick, Andrew Whitworth, Nate Tice and Andrew Phillips.
Advertisement
The panel of experts reviews the film of the model making predictions and makes sure it’s identifying legitimate threats and not looking at players who could not be rushers to the well-trained eye. Some of their feedback, along with that of Schwartzstein, who provides feedback on every play, is fed back into the system.
Even when the model isn’t right on a blitzer, it’ll end up highlighting a player who is doing something out of the ordinary. There was a play Thursday when the Chiefs had two quarterback spies on Russell Wilson. They didn’t blitz, but the model highlighted them because the movement pattern told it they were doing something different.
Overall, the model is predicting blitzes at a very high percentage, but when it does miss one, the defense is likely doing something unique with a movement pattern that it hasn’t picked up before. The defense could be doing a very good job of disguising its intentions, but the model will record it so it doesn’t get fooled next time.
![go-deeper](https://cdn.theathletic.com/cdn-cgi/image/width=128,height=128,fit=cover,format=auto/app/uploads/2023/10/17000117/1017Cover7TuesdayRyans-1024x683.jpg)
GO DEEPER
NFL’s best and worst Week 6 coaching decisions: DeMeco Ryans keeps Saints guessing
Schwartzstein said the idea for Defensive Alert stems from his time in college when he was already obsessed with the idea of machine learning and football. With his partnership with Amazon, he has the tools and backing to turn his ideas into reality.
“Betsy Riley, my boss, Alex Strand, Jared Stacey, Marie Donoghue and Jay Marine were challenging me to think bigger,” Schwartzstein said. “Because we do have a great machine learning team that’s international and you have access to them, what can you do? When we first started, we looked at making the predictions rules-based: Like what I would do if I was the center or QB? How many eligibles do we have versus how many defenders are on that side of the field? Field pressure is likely where we’re going to set the prediction. Where are we on the field? How close are the defenders? But then, as we got closer, we thought, let’s hand this over to the machine learning team and have them use an AI machine learning model to solve this problem.”
The international team wasn’t familiar with American football, so Schwartzstein had to educate them on the different positions and aspects of the game like motion and how it affects the defense.
“The thing is, these scientists are unbelievable — so this is cool for them because football is a very data-heavy sport with many different data sources you can plug in. They’re so willing and so hardworking at getting this to the next level. And now I’m having arguments with some of them about what teams are doing in certain scenarios. They’re talking about their fantasy teams because they’re wanting to get more invested into the sport.”
Advertisement
Just to ease the minds of concerned fans, teams cannot use this model to their advantage in games. Communication with the quarterback is cut off after 15 seconds of the play clock has expired and there’s no way to get information to the quarterback fast enough. Also, coaches in the booth don’t have access to the Amazon broadcast. Technology usage is extremely restricted for teams. They don’t have access to tracking data during games, and even when they are looking at their tablets, they are looking at stills, not video.
Not every viewer will choose the Prime Vision option. Some prefer to watch the game as they always have. At first, I loved the idea of Prime Vision because it allowed me to watch the game from a wide angle in which I could see all 22 players at once. How do you know what’s happening when you can’t see the secondary? Regular broadcast angles cut them off to zoom in on the QB. But with the added features that Schwartzstein and his team are including like Defensive Alert or Prime Target, which shows viewers which receivers are getting open with a green circle, viewers can watch the game unfold like never before. Viewers don’t have to rely on commentators to tell them the story of the game, they can see it for themselves.
![go-deeper](https://cdn.theathletic.com/cdn-cgi/image/width=128,height=128,fit=cover,format=auto/app/uploads/2023/10/18232453/Cover7-10192023_Thursday-1024x683.jpg)
GO DEEPER
Rating every NFL starting QB so far this year. Who’s better than expected? Most concerning?
(Top photo courtesy of Amazon Prime)
“The Football 100,” the definitive ranking of the NFL’s best 100 players of all time, goes on sale this fall. Preorder it here.