NFC West: Total Quarterback Rating

QBR: Warner, Hasselbeck and playoffs

August, 19, 2011
8/19/11
4:53
PM ET
The NFC West discussion on QBR continues upon request.

"I wonder what Kurt Warner's QBR was during the Cardinals' run toward the Super Bowl, and the ridiculous offensive display against the Packers the following year," Facebook friend Kevin wrote.

Others had similar thoughts regarding Matt Hasselbeck's postseason production.

"The crowds are clamoring," Peter wrote, also via Facebook.

Among the findings, courtesy of ESPN Stats & Information:
  • Aaron Rodgers' lights-out performance against the Atlanta Falcons in January last season produced the highest QBR score (97.2 out of an impossible-to-attain 100) during playoff games over the last three seasons. Rodgers completed 31 of 36 passes for 366 yards. He accounted for 18 first downs passing and one more rushing. He had eight third-down conversions and supplemented his three scoring passes with a rushing touchdown. Jon Gruden, who helped shape criteria for QBR, called this one of the two best playoff performances from a quarterback he had seen. Steve Young's six-touchdown Super Bowl against San Diego was the other.
  • Warner's performances against the Packers (2009 season) and Eagles (2008) rank second and third, respectively. Warner had more touchdown passes (five) than incomplete passes (four) against the Packers. He had nearly as many scoring passes (four) as incompletions (seven) against the Eagles. Warner wasn't a scrambler, but he was often at his best in big games, and in big moments of those big games. Even after tossing a killer interception before halftime against Pittsburgh in the Super Bowl, Warner came back with a go-ahead touchdown pass in the final minutes.
  • Hasselbeck's performance against New Orleans in the wild-card round last season, though impressive, did not threaten the top QBR scores. His QBR score was 78.2 for this game. The top six scores were all in the 90s among the 63 qualifying performances (minimum 25 action plays). The 78.2 score was a good score, but none of the quarterbacks with postseason QBR scores in the 90s tossed an interception. Hasselbeck did against New Orleans, on a third-and-1 play in his own territory. Marshawn Lynch gave Seattle the cushion it needed late in the game. Seattle threw no passing touchdowns in the final 18 minutes.
  • Hasselbeck tossed three touchdown passes during a lopsided defeat against Chicago the following week. His stats were good enough to produce a traditional passer rating of 94.3, but the QBR score was only 38.3 because the Seahawks fell behind by a wide margin and never threatened. His scoring passes covered only 2, 3 and 9 yards. Hasselbeck's stats were inconsequential. QBR was not impressed.
  • Jay Cutler's playoff performance for Chicago against Seattle last season made the list thanks in part to his two rushing touchdowns and five overall rushing first downs.

The chart shows the six highest postseason QBR scores since the 2008 season (minimum 25 plays).

A conventional stat sheet cannot tell us why Tarvaris Jackson's effort against Buffalo last season held up so well despite three interceptions, including one returned for a touchdown.

Dean Oliver can.

Oliver, production analytics director for ESPN Stats & Information, pointed to one play I missed when explaining why Jackson's performance against the Bills beat out all but six others by Kevin Kolb, Matt Hasselbeck, Sam Bradford and Alex Smith last season.

QBR recognized Jackson's end-zone pass for Bernard Berrian as significant for drawing a pass-interference penalty. The penalty turned third-and-2 from the 6-yard line into first-and-goal from the 1. Adrian Peterson scored the go-ahead rushing touchdown two plays later.

Jackson led the Vikings to a touchdown on their next possession by passing or running for all three first downs on the drive, including one in the red zone. Jackson then threw a touchdown pass to Sidney Rice on the Vikings' next possession, blowing open the game, 28-7.

The two interceptions Jackson subsequently threw didn't matter so much, although they dragged down his conventional passer rating to 85.0 on 158.3-scale.

I've reproduced the chart from the previous item. It shows the 10 highest single-game QBR scores from 2010 for the quarterbacks in question (minimum 20 action plays per game).

Jackson's mobility accounted for some of his appeal from Seattle's perspective. QBR takes into account quarterback rushes, among many other factors. If it invites closer examination of quarterbacks' performances beyond the obvious, all the better.

QBR: Kolb, Bradford and the NFC West

August, 18, 2011
8/18/11
11:15
AM ET
EARTH CITY, Mo. -- Watching Sam Bradford fire passes across the St. Louis Rams' practice field has affirmed his status in my mind as the most promising quarterback in the NFC West.

There's much to like about the Arizona Cardinals' Kevin Kolb based on first impressions.

[+] Enlarge
Sam Bradford
Icon SMISam Bradford had two of the five best-performances by a quarterback last season in the NFC West, according to QBR.
The focus on potential should not entirely overshadow actual in-game performances, of course. With that in mind, and in response to several of your questions, I've acquired from ESPN Stats & Information single-game QBR data for current NFC West passers. Data goes back to 2008, but I singled out 2010 games as most relevant, narrowing the list further by considering only performances in which a quarterback participated in at least 20 action plays.

The chart ranks the 10 best QBR performances last season for games featuring Kolb, Bradford, Alex Smith, Tarvaris Jackson and Charlie Whitehurst. I also included Matt Hasselbeck's games given that Seattle, San Francisco and Arizona considered signing him this offseason.

Whitehurst made only two starts, and neither cracked the top 10.

Jackson, ahead of Whitehurst on the Seattle Seahawks' depth chart, started only one game, for Minnesota. Two of his performances qualified for consideration, and one of them, in relief against Buffalo in Week 13, cracked the top 10.

Jackson tossed two touchdowns with three interceptions in that game. He took only one sack, completed a 31-yard touchdown pass, found Sidney Rice for a 46-yard gain on third-and-11 and scrambled for a 13-yard gain to help set up another score. The Vikings won, 38-14.

I'm still getting a feel for QBR after years of reliance on more basic passing-only stats. As game situations become less critical -- say, early in the game or when the score is lopsided -- QBR gives less weight to specific plays. A quarterback playing well enough to build a big lead won't see his QBR suffer as much if he simply manages the game from that point forward.

In looking at all NFC West performances from last season, I noticed Troy Smith's Week 12 showing during a 27-6 Monday night victory over Arizona rated higher than his rollicking Week 10 showing during an overtime victory against St. Louis.

Smith's pure passing stats against the Cardinals weren't nearly as good as they were against the Rams. His passer rating was 55 points lower against Arizona (61.7) than it was against St. Louis (116.7). But QBR resoundingly favored Smith's Arizona performance, 73.5 to 49.2.

Against Arizona, Smith scrambled for seven yards on third-and-4. He threw a 38-yard touchdown strike to Michael Crabtree in which the ball traveled the full 38 yards in the air, more impressive than if Crabtree had racked up yards after the catch. A 16-yard completion on second-and-10 sustained another scoring drive as the 49ers built a 14-3 lead. Smith's rating peaked during this drive at 97, falling only gradually the rest of the game as the 49ers built a 21-3 lead and coasted to victory.

And now, on with the list. Hasselbeck accounted for four of the 10 highest qualifying QBR performances (there were 50 total). Bradford made the top 10 list three times. Kolb had five qualifying performances and one of them, against Atlanta in Week 6, topped the list.

Hasselbeck also accounted for six of the 18 lowest-qualifying QBR performances. Bradford accounted for five of them.

The five worst overall: Bradford (2.1 QBR) at New Orleans in Week 14; Hasselbeck (2.7) against Atlanta in Week 15; Whitehurst (4.5) against the New York Giants in Week 9; Whitehurst (6.3) against Tampa Bay in Week 16; and Hasselbeck (8.5) against Oakland in Week 8.
Mike Greenberg learned his lesson picking the San Francisco 49ers to reach the Super Bowl last season.

The "Mike & Mike in the Morning" co-host has penciled in the 49ers for another 6-10 season after the team re-signed Alex Smith as its likely starting quarterback and parted with several defensive starters. Co-host Mike Greenberg has the 49ers doing slightly better, but both think the team would have been much better off signing Matt Hasselbeck instead of Smith.

The problem: Hasselbeck wasn't available to San Francisco before the lockout. The 49ers could not know for sure whether Hasselbeck would sign with them. They knew Hasselbeck had stronger ties to the Tennessee Titans, and to the Seahawks, of course. The 49ers did not know when the lockout would end. In the meantime, Smith was available to them at a reasonable price.

Smith and Hasselbeck were coming off similar seasons, according to QBR. That evidence and pure passing stats do not demonstrate beyond a reasonable doubt that the 49ers would be far better off with Hasselbeck as a one- or two-year rental. But anyone watching Hasselbeck destroy New Orleans in the playoffs last season knows Hasselbeck can, at his best, play at a far higher level than Smith has achieved during six disappointing seasons with the team.
Total Quarterback Rating (QBR) should affirm what we know most of the time while occasionally challenging what we think.

The newly developed metric would have no value if it tried to convince us Derek Anderson was better than Tom Brady (it does not do this). If it told us Chad Pennington was better in 2008 than Peyton Manning was in 2010 -- it does make that case -- we should at some point revisit those specific seasons for those specific quarterbacks. I've been most interested in learning what QBR reveals about performances relative to the passer-rating formula in use since 1973.

For example, Matt Hasselbeck finished with similar passer ratings in 2009 and 2010, but QBR favored his 2010 season by a wide margin. Something about Hasselbeck's 2009 season did not sit well with the QBR formula. Even his 2008 season, complete with a career-low 57.8 passer rating, held up better. Why was this?

Alok Pattani from our analytics team pointed to a couple of factors:
  • Hasselbeck made greater contributions as a runner in 2008 than in 2009.
  • Hasselbeck fumbled only once in 259 plays during the 2008 season. Seattle recovered. He fumbled 18 times in 1,139 plays during the 2009 and 2010 seasons. Seattle lost eight of those 18 fumbles. That would have helped 2008 make up ground.

There were other factors, but those two were prominent. Separately, Hasselbeck's production on third down improved steadily since 2008. His third-down completion percentage rose from 36.7 in 2008, to 53.6 in 2009 and 60.9 last season. His sacks per pass play also steadily declined on third down.

I was also curious why Hasselbeck's QBR for 2010 doubled the 2008 QBR for then-San Francisco 49ers quarterback J.T. O'Sullivan, even though their passer ratings were similar. One key: O'Sullivan wasn't nearly as good on third-down pass plays across categories that included completion percentage, touchdowns, interceptions, sack percentage and conversion rate.

Understanding such disparities requires understanding QBR. Once QBR becomes established, considering the computations will not be relevant for those merely interested in the bottom line. We'll simply need to know that a season-long QBR around 50 would be near average, while a season-long QBR in the 65 range would reflect Pro Bowl-caliber play. Players will rarely reach an 80 QBR for a season, and it's impossible to reach 100 because a quarterback could, in theory, always complete one more pass for one more yard, etc.

ESPN plans to unveil details more fully during a "SportsCenter" special Friday night at 8 p.m. ET. A news release promoting the new metric offers the following details regarding QBR:
  • Total QBR measures all of the significant contributions by a quarterback during the course of a game and accounts for precisely how much he impacts his team’s performance and chances of winning.
  • Total QBR is based on all of a quarterback’s plays (rushing, passing, sacks, fumbles, interceptions, penalties, etc.), and it calculates the per-play net impact of the quarterback on the ability to score. Each play is weighted by the situation (i.e., down and distance, field position, time during the game) and its importance to the game’s outcome.
  • Another variable, division of credit, assigns a percentage to how much credit a quarterback should get for a positive play -- or blame for a negative play. ESPN analyst Trent Dilfer helped come up with additional data to consider, including how far a pass travels in the air, where the ball was thrown on the field, yards after catch, whether the quarterback faced pressure, etc.
  • QBR draws from 60,000 plays over the past three years to assign credit or blame for every play involving the quarterback. It measures quarterbacks on a 100-point scale, whereas the NFL's passer rating formula maxes out at 158.3.

According to those standards, Kurt Warner is the only NFC West quarterback since 2008 to post a Pro Bowl-caliber season.

And now, on to the chart showing passer rating and QBR figures for NFC West quarterbacks since 2008.

The official NFL stat sheet said Kurt Warner played a perfect game for the Arizona Cardinals against Miami back in 2008.

Total Quarterback Rating (QBR) knew better.

While Warner was brilliant that day, completing 19 of 24 passes for 361 yards and three touchdowns, his performance could have been statistically superior. He scored a 97.7 on the 100-point QBR scale.

Warner took two sacks for 12 yards in that game, affecting his QBR. The 75-yard touchdown pass he threw to Larry Fitzgerald traveled only 17 yards in the air. About half his passing yardage that day came after the catch.

QBR, set to debut during a "SportsCenter" special Friday night, keeps moving the carrot as quarterbacks chase perfection. Albert Larcada, ESPN Stats & Information analytics specialist for the QBR project, illustrated how this happens by analyzing the three performances since 2008 featuring 158.3 ratings.

Drew Brees posted a 158.3 rating against New England during the 2009 season. That effort translated to 98.6 by QBR standards. Brees took one sack for 4 yards. The 18-yard touchdown pass he threw to Pierre Thomas featured 25 yards after the catch. (Thomas caught it behind the line of scrimmage.)

No matter how well a quarterback plays from a statistical standpoint, he could have fared better.

Completing all 20 pass attempts, each for a 99-yard touchdown, would shatter records. But the performance wouldn't rate as high as one featuring 21 touchdown passes of that length. And so on.

That's why it's misleading to say a quarterback played a "perfect" game when his passer rating maxed out at 158.3 under the formula in place since 1973.

Tom Brady's 158.3 rating against Detroit last season translated to 94.6 in QBR. Brady's receivers made huge gains after the catch in that game, to a degree much greater than they would have done typically.

In theory, a perfectly executed short pass could free up a receiver for additional yards.

"It’s true a perfect pass could set up additional air yards for a receiver," Larcada explained, "but on average we found YAC to be mostly on the receiver, and pretty strongly so. Good receivers get more YAC per completion than bad receivers. Good quarterbacks don’t necessarily get more YAC per completion than bad ones."

More YAC means fewer air yards when all else is equal, hurting a player's QBR number.

"Even with a perfect pass, the receiver still must have the speed/moves/quickness to create the additional yardage," Larcada said.

QBR takes into account many more variables. It grades each play in relation to how it affects game outcomes, putting more weight on a killer interception than a meaningless one on a Hail Mary at the end of a half.

We'll continue the discussion Thursday.
Unsettled quarterback situations elsewhere in the NFC West made the St. Louis Rams a logical early favorite to win the division in 2011.

The Rams had the very promising Sam Bradford, after all, while the rest of the West was going through quarterback identity crises.

Bradford
Bradford
Smith
Extensive field testing has shown that peppering a well-informed San Francisco 49ers fan with such logic can produce entertaining (for the rest of us) results. The embattled 49ers fan, no matter how repulsed by the thought of another season with Alex Smith behind center, will sometimes retrieve from his statistical weapons cache this improbable bombshell: Smith was the highest-rated passer in the division last season.

And it's true. Smith finished the 2010 season with a borderline-respectable 82.1 rating, even though his team struggled to a 6-10 record. Bradford's rating was 76.5, with Matt Hasselbeck at 73.2. But what if we viewed these players' contributions through the new "Total Quarterback Rating" (QBR) tool set to launch for the upcoming season?

I do not yet have all the details on how the formula works -- ESPN will discuss those in depth during a "SportsCenter" special Friday -- but I did secure QBR numbers for a few quarterbacks relevant to this discussion. QBR evaluates quarterbacks on a 100-point scale, reflecting how a quarterback's performance on each play affects game outcomes.

In preparing this item, I asked ESPN Stats & Information for examples of quarterbacks with similar passer ratings and disparate QBR numbers. We might then better illustrate how passer rating differs from QBR. Smith and Tennessee's Kerry Collins provided one such example. Their passer ratings were nearly identical, but Collins scored much higher in QBR.

Collins' 56.0 rating in the new metric was six points higher than the average established by all quarterbacks across roughly 60,000 plays since 2008. Smith was well below average in QBR at 40.0, affirming what I suspect most of us intuitively know about these quarterbacks. In this example, the QBR seems quite helpful.

Smith took sacks at a higher rate than Collins. They threw interceptions at a nearly identical rate, but the QBR formula considered Smith's interceptions far more costly to his team, according to Alok Pattani, one of two analytics specialists assigned to the QBR project. QBR also viewed Bradford and Hasselbeck more favorably than it viewed Smith.

In another example, QBR rated Peyton Manning much higher than it rated Matt Schaub last season, even though Schaub finished with a slightly higher passer rating (92.0 to 91.9). Schaub took twice as many sacks while fumbling three times as frequent.

Manning's QBR was 69.5, right around where we might expect a Pro Bowl-caliber quarterback to rate. Schaub's QBR was 57.8, above average but not as high as his passer rating ranked in relation to other quarterbacks.

On Bradford, I'm interested in knowing more about how his depleted receiving corps affects his standing in QBR. While he was better than Smith in QBR, I anticipated a greater margin. Bradford ran out of weapons at receiver and leaned heavily on short passes with little potential. Likewise, the challenges Seattle faced on its offensive line put Hasselbeck at a severe disadvantage, I thought. These are all things to explore once QBR makes its formal debut Friday.

The late Don Smith never claimed his passer-rating formula was perfect.

Quite the opposite, in fact.

[+] Enlarge
Tom Brady
Al Bello/Getty ImagesAccording to an outline for the rating system, Tom Brady would fall in the "top tier" category.
"Some people call it a quarterback rating system, but that really is not what it is," Smith told me during a 2002 interview. "It’s simply a passing statistic."

I've actually defended Smith's rating system because the quarterbacks with the highest ratings -- Tom Brady, Philip Rivers and Aaron Rodgers led the way last season -- usually are the best quarterbacks. But there's so much more to quarterbacking than passing stats for touchdowns, interceptions, attempts, completions and yardage.

Game situations should count for something, and now they do.

With input from football people, including ESPN analyst Trent Dilfer, our statistical analysts have developed a 100-point ratings scale for quarterbacks taking into account advanced stats, game situations and relevant non-passing stats, including fumbles and sacks, to evaluate quarterbacks far more thoroughly. The methodology is complex -- one of the formula's key algorithms spans some 10,000 lines -- but the resulting "Total Quarterback Rating" (QBR for short) beats the old passer rating in every conceivable fashion. The ratings scale will debut this season.

I've been bugging the Stats & Information team for a sneak peak ever since learning former NBA statistical analyst Dean Oliver had joined our production analytics unit and was playing a prominent role in QBR development. Oliver, a Caltech grad with a Ph.D. in statistical applications, revolutionized how NBA teams use advanced statistics. Menlo College professor Ben Alamar, who has consulted with the San Francisco 49ers, is also part of the team.

Our stats team has been using game video to track stats relating to pressure, personnel, formation, game situation and more since 2008. The QBR stat represents a significant leap in harnessing those statistics for something more.

The old formula Smith created treated stats the same regardless of circumstance. A touchdown pass thrown against a prevent defense during a blowout defeat equals one thrown against pressure to win the game. A 5-yard completion on third-and-4 counts the same as a 5-yarder on third-and-15. A critical quarterback scramble, sack or fumble doesn't even factor.

"There is no way to statistically say how effective a guy is under fire," Smith lamented during our 2002 conversation. "None of that can be put into something like this."

Now it can, along with a whole lot more.

The QBR formula takes into account down, distance, field position, time remaining, rushing, passing sacks, fumbles, interceptions, how far each pass travels in the air, from where on the field the ball was thrown, yards after the catch, dropped balls, defensed balls, whether the quarterback was hit, whether he threw away the ball to avoid a sack, whether the pass was thrown accurately, etc. Each play carries "clutch weight" based on its importance to game outcome, as determined by analyzing those 60,000 plays since 2008. The stats adjust for quarterbacks facing an unusually high number of these situations.

"If it is a running clock late in the game, maybe you only get a few yards here or there, that is the right football play to make," Jeff Bennett, senior director of ESPN's production analytics team, said Sunday. "We spent a month learning about ratings to make sure quarterbacks couldn’t game the system, so they're not afraid to throw that deep pass at the end of the first half and risk an interception."

I've seen an outline for the rating system breaking down 2010 quarterbacks into six general categories, from top tier to poor. Precise rating numbers were not yet available. The quarterbacks under consideration broke down as follows:
ESPN plans to enlist several quarterbacks when introducing the stat during an hour-long "SportsCenter" special Friday at 8 p.m. ET. We'll be referencing the stat on the blogs and elsewhere. Bennett said he's allocating enough manpower to produce ratings on game days, a huge help for those of us analyzing player performances shortly after games.

"We want to reward a good football play," Bennett said.
BACK TO TOP