Not adding up: Brian Schottenheimer’s rule of 53

Charles Reep is known as both the founding father of soccer analytics and the man who “helped ruin decades of English soccer.”

Beginning in the 1950s, Reep published a series of papers examining the sequences that lead to scoring a goal. Reep found that most goals were scored after a series of three or fewer passes, which led to his recommendation: Teams should avoid long passing sequences by kicking the ball far down the field and trying to chase it down. Eventually, this advice became a cornerstone of English coaching.

The problem? Possession is fluid in soccer. Most goals were scored after short sequences because the vast majority of possessions were short sequences. When one accounts for the prevalence of each sequence length, it turns out that maintaining possession of the ball is associated with an increase in the likelihood of scoring a goal. The error in logic is akin to stating that motorcycles are safer than cars because there are fewer fatal accidents involving motorcycles, ignoring that motorcycles constitute a small percentage of vehicles on the road.

In other words, Reep’s advice was exactly backward. Reep’s contention that “passing for the sake of passing can be disastrous” was misguided.

We now turn to the Seahawks, whose actions suggest that they view passing with the same disdain that Reep did. And — like English soccer during the long-ball era — the Seahawks’ devotion to running the football appears to be based on bad math.

In his profile of Brian Schottenheimer published last August, The Seattle Times’ Bob Condotta explained the numbers behind Schottenheimer’s prioritization of the run game: “Schottenheimer says the real key to his offensive philosophy is rooted in the magic number: 53. In his career, Schottenheimer has found hitting that number through any combination of pass completions and runs in a game is one of the biggest predictors of victory.” Condotta added a quote from Schottenheimer: “We are always going to go in with the understanding that we should be able to run the ball, and believing and expecting we can do it at a high level.”

For the remainder of this piece, we’re going to look at the problem with the “rule of 53.” As with the English soccer example, the rule illustrates the perils of making a decision based on a correlation without understanding why the correlation exists.

Hocus-Pocus

First, let’s take a look at the magic number itself. Where did 53 come from? If it were a true magic number, we’d expect a jump in win percentage once teams hit that number. Is that the case? Below is the relationship between the number of completions plus rush attempts and win percentage (as always, all numbers cover the 2009-18 seasons using data from nflscrapR).

Without looking at the horizontal axis labels, one wouldn’t be able to guess which dot represents 53 completions plus rush attempts — there’s no discrete jump anywhere in the figure. In other words, there’s nothing special about the number 53. This shouldn’t be surprising given that Pete Carroll and Tom Cable believed that 50 was the magic number as recently as 2017, and Bill Parcells named 51 as his magic number.

As a broader point, we should be skeptical of anyone claiming to have found a magic number. The following is from a piece published on the Seahawks’ website discussing analyst Mike Eayrs’ influence on Pete Carroll:

“When Eayrs looked at what it took to increase the chances of scoring or preventing a score on any given possession, he found that the magic number was 12 yards for a running play and 16 yards for a passing play, numbers the Seahawks and many other teams still use to define explosive plays.”

Magic number = completions + rush attempts

So far we have discovered that the magic number of 53 is completely arbitrary. But there’s still a positive relationship between how many completions plus rush attempts a team compiles and the likelihood it wins a game. So why is targeting this number problematic?

To investigate what is driving the positive slope in the figure above, let’s plot the relationship between win percentage and rush attempts (left) and between win percentage and completed passes (right).

When we break down the magic number into its components, neither of these relationships should be surprising. On the right, we have a variant of the Passing Paradox, a term coined by ESPN analyst Brian Burke more than a decade ago, which states that the worse a team is at passing, the more it’s forced to do it (because teams fall behind in games and need to pass more to try to catch up). The lesson isn’t that passing a lot is inherently bad but that the number of pass attempts is in part a reflection of game state.

And on the left, we see that rush attempts and win percentage are highly correlated, the relationship that drives the Run To Win fallacy.

The Run To Win fallacy

Now we’re back to familiar territory. One of the earliest contributions of the football analytics movement was debunking Run To Win, the idea that the positive correlation between rush attempts and winning means that teams should make an effort to run the ball more throughout the game. In fact, Football Outsiders was founded in 2003 to negate this very idea. In the introductory piece that launched the site, founder Aaron Schatz wrote that “more rushing attempts early don’t indicate a winning team, but rushing attempts late do.”

Here is a visual representation of that finding:

In the first quarter, there isn’t much of a relationship between rush attempts and team success. But in the fourth quarter, the sample of teams that run the ball at high frequencies consists of those that have already built up a lead and are milking the clock, while the teams that don’t run much are facing a deficit and frantically trying to catch up.

The problem with citing the correlation between rush attempts and wins as a reason to prioritize the run game is that the relationship exists only for teams that have already built a lead. This advice gives no actionable information about the hard part: getting the lead in the first place. The correlation between rush attempts and wins is just as useless as the correlation between quarterback kneels and wins. Both contain the same information: Teams that have a lead late in the game tend to end up winning.

However, running at high rates early in the game does not help teams get a lead late in the game.

The above is from Pro Football Focus’ Timo Riske, comparing the run rates on first-and-10 of teams who ended up winning a given game against run rates of teams that would go on to lose. Riske found that teams running more early in the game are less likely to go on to win, a finding that has since been replicated by FiveThirtyEight’s Josh Hermsmeyer.

Just like the English soccer example, the lesson taught by the rule of 53 is exactly backward. Bad math has driven bad decision-making, with detrimental results on the field. In Brian Schottenheimer’s debut 2018 season with the Seahawks, Seattle was the most run-heavy team in the entire league after accounting for game state despite having an elite quarterback in Russell Wilson.

The culmination of Seattle’s run-heavy strategy came in its playoff game in Dallas, when on first and second down in the first three quarters, the Seahawks called 19 rush attempts to only 10 dropbacks despite having 0.36 Expected Points Added (EPA) on those dropbacks, compared with -0.16 EPA per play on rush attempts. Despite the defense holding the Cowboys to 10 points through three quarters, the Seahawks held just a four-point lead going into the fourth and went on to lose.

Correlation is not causation

In Seth Partnow’s rundown of teams making these fundamental mistakes in soccer, basketball, and hockey, he wrote that “three-pointers and shots at the rim are indicators of good offense, and not necessarily good offense in and of themselves — less a cause than a result.”

A similar logic applies to rush attempts. Compiling a high number of rush attempts is typically the result of having a lead late in the game, but not the cause. Figuring out the cause requires answering the crucial “why” question: Why are most goals scored after short passing sequences? Why do winning NFL teams have a lot of rush attempts?

Acting without understanding the why is when teams can be led astray.

(Top photo: Ted S. Warren / Associated Press)