Production Above Replacement

by Will Roberts

Now that you (hopefully) understand Production Score and how it is used to measure player performance, we can begin to look more closely at evaluating and comparing players. When looking at total Production Score output, it is not uncommon to find quarterbacks at the top of the list in a given game or season. This probably doesn’t come as a surprise to most of you reading this since quarterbacks are often the focus of front offices, coaches, and media alike. And by looking at total Production Score, it’s for good reason. Quarterbacks tend to be responsible for or involved in the most touchdowns of any position. In large part because they are almost always involved in any sort of passing touchdown and can score using their legs too. So it’s only natural that they would have the highest Production Score since the model is based on frequency to touchdowns.

The question becomes how do we fairly compare a player’s production to other positions? Trying to compare a running back to a quarterback based on total Production Score alone would be a little too apples-to-oranges. One common way to do this would be to only compare a player to peers at the same position. We could still gain meaningful insights from their Pass Production, Rushing Production, or Receiving Production, so we might be able to answer who was the best at that position for a given week. But we still need to find a way to compare across positions to determine who is the cream of the crop.

One way to attempt to provide an answer would be to look at a player’s relative performance compared to his position. By looking at how much a player produced against his position’s average, or better yet, median Production Score, we can get a better sense for their performance relative to what could be expected from their position. I call this Production Above Replacement, or PAR, and we can use this as a starting point to measure performance. The term replacement isn’t meant to be a slight toward players or indicate that they are easily replaced, rather it’s simply what we would expect a middle of the pack player at each position would produce. In most people’s minds, this is what they mean when they use the term average. But by instead using the median, we know that 50% of the players will fall above the measure and 50% will fall below…. Quite literally the middle of the pack. Using the median also handles skewed data and outliers much better than the true average or mean of the data, which we are bound to come across over the course of a season. If talk of median and percentages makes your head spin, I hope an example might illustrate how these new metrics can be helpful.

Let’s say your favorite wide receiver, Sticky Hands, had a Production Score of 6.7 for a given week of the NFL season. Let’s further assume that the median Production Score of wide receivers that same week was 5.0. We can find Sticky’s PAR by taking the difference between his Production Score and the median production score, or 6.7 - 5.0 = +1.7. But what does this +1.7 actually mean? Couldn’t we just use PAR to compare across positions? Mmmmm……..Not really.

Depending on the data, a PAR of +1.7 could be a meaningful Production Above Replacement, or it could be just slightly above the median, making Sticky pretty much middle of the pack. This is where looking at the percentile rank of Sticky’s Production Score can give us some much needed perspective.

One of the easiest ways to think about percentile rank is to answer the question “What percentage of the values fall at or below the value I’m looking at?” An easy example would be to look at the median value, since we know 50% of the values fall at or below this by definition. This means that the median is also the 50th percentile, every time! By looking at the percentile rank of Sticky’s PAR, we’ll be able to determine how well he played in that game. I’ll skip the short math lesson on normal distributions, standard deviation, and z-scores that tell us how to calculate percentile rank and jump right to why we bother even looking at percentile rank in the first place - comparing across the different positions.

Sticky’s Production Score tells us roughly how many points his production is worth, his PAR tells us how much more production than a middle of the pack receiver he provides, but it is the percentile rank of Sticky’s PAR that allows us to compare him to other positions. Sticky’s PAR of +1.7 could put him in the 80th percentile for receivers, but a PAR of +1.7 might only be in the 55th percentile for quarterbacks as an example. In this case, both players have an identical PAR, but we could make a very strong case that Sticky played the better game, since he played better than 80% of his peers!

Percentile rank can also help us control for different time periods. For example, Sticky’s PAR of +1.7 could be great for a single week, but over the course of a season, a PAR of +1.7 may gradually move closer and closer toward the median as the cumulative total of Production Scores grow. As more production occurs throughout the season, a PAR of +1.7 becomes a smaller and smaller piece of a larger pie. In order to maintain his percentile rank, Sticky would need to continue producing at a high level. We could also look at PAR as a rate per game, or PAR/game, to help us control for injuries or missed games.

It is my hope that this quick read has helped you understand the methodology behind PAR and how we can use it along with percentile rank to compare our favorite players regardless of what position they play. Maybe now we can settle the debate of who played the best once and for all!