As one of Baseball Prospectus' flagship stats, VORP (Value Over Replacement Player) is probably best known to the general public as  some nerdly invention that infuriates crusty old deans sports journalists like Murray Chass and Jon  Heyman. "The" internet responded in predictable (if predictably hilarious) fashion, in blog posts undoubtedly typed with 'faces born of contempt.'

I'm not writing to continue that tired non-debate or to make fun of those crochety ol' "mainstream media" types. Rather, I'm going to review some legitimate criticisms of VORP for position players (there are other issues for pitchers) and to see what sort of practical weight the problems of VORP might have, that is, does it matter beyond just a "few runs" that internet sabermetricians are carping about?

In an earlier column, I discussed and utlized Justin Inaz's Total Value spreadsheet for 2008, a very carefully constructed "überstat" for evaluating individual players. There are many cool things about the spreadsheet that I discuss in the earlier article. You can also find it referenced a fair bit elsewhere on the 'Net.

One prominent use of Justin's numbers is Sky Kalkman's series at Beyond the Box Score reviewing the best players of the 2008 season at each position. What's even cooler is that Justin has allowed Sky access to his spreadsheet. Using the data, Sky put together a "JustVORP" spreadsheet. Basically, Sky just leaves Justin's defensive stuff (well-done in its own right) out, and gives a list of each players positionally-adjusted runs above replacement level. Thus, it neatly parallels Baseball Prospectus's VORP (created by Keith Woolner), which also does not include defense, but gives the positionally-adjusted offensive runs above replacement for individual players.

I am certainly not dismissing defense (it's the 'New Moneyball,' after all). However, since  there is so much disagreement about how defense should be evaluated (whether or not stats should be used at all, and if so, which one), that using stats like VORP (and JustVORP), which give everything "but" defense, allows the user to then figure in the defensive adjustments in his or her own fashion when evaluating a player.

Just to make this clear again: I am not saying that we should forget about defensive stats. I am simply leaving them aside for the purposes of this column -- to see how much VORP does or doesn't go wrong in evaluating a player's relative offensive contributions. Along the same lines, this shouldn't be taken as an endorsement of postionally adjusted offensive stats taken on their own. I am very much against a "this player doesn't hit well enough to be a corner outfielder" sort of thinking which doesn't taken defense into account -- see Dave Cameron's excellent write-up on that topic. I'm using position-adjusted offensive stats here just to see where VORP goes wrong, considered on its own terms.

So what are the differences between JustVORP and VORP? Why prefer one to the other? Sky summarizes it well in his JustVORP post, which you should read, but I will summarize some of  the issues briefly:

• BP's VORP uses Marginal Lineup Value as its base run estimator. As has been shown in various places, MLV undervalues walks by half. That's right, Baseball Prospectus, the avant-garde of the OBP revolution (well, maybe 40-50 years after Branch Rickey), have the value of walks wrong in one of their flagship stats. Justin's stats, on the other, hand, use BaseRuns (discussed in many helpful places -- see Colin Wyers' "Batman" series listed at the end of this article for one) to establish the correct linear weight of each event. In short, BaseRuns is a dynamic run estimator that allows for the changing run expectancy of each baseball event in different run environments -- even the most extreme, where even linear weights break down (as for Runs Created and MLV, one might say they were broken to begin with...).
  
• JustVORP adjusts for the superiority of AL pitching, VORP does not.
• VORP defines replacement level in relation to the league offensive average at each position for the season. This has the weird result of making DHs seem more valuable than 1Bs, because their averages aren't hitting well. Yet we know that DH is the easiest position to "replace," and is thus (all things being equal) the least valuable. Just because teams aren't using their DH spot doesn't change that fact. That's just the most obvious example -- in some seasons, the average CF hits better than the average LF (think of teh 1950s). But that certainly doesn't mean that the LFers are harder to find and replace, because all CFers can play LF, but not vice-versa. VORP (in some eras) might lead one to think otherwise. JustVORP avoids this by using Tom Tango's positional adjustments (prorated for playing time, of course): Catcher +12.5, SS +7.5, 2B/3B/CF +2.5, LF/RF -7.5, 1B -12.5, DH -17.5.
  

But what is the practical effect of all this? Is this just sabermetric nitpicking? In one discussion on the Book Blog, comparing VORP with BP's RARP (runs above replacemente using Equivalent Runs -- a better run estimator anyway that approximates the correct linear weights), Colin Wyers found that RARP itself revealed VORP to be undervaluing not only players who walked a lot, but also people who hit a higher than usual proportion of doubles. The average VORP "error" along these lines for regulars was about 5.8 runs, and about 6.6 for the doubles hitters. That's a pretty big error, once you realize that that's more than half a win -- and teams are pahing between four and five milllion dollars for each marginal win in 2009. And that's just addressing the problem in terms of the linear weights.

What does this come to in concrete terms? For the sake of discussion, let's assume that JustVORP is on the right track in terms of run estimation, positional adjustment, etc. How big of a difference does just using plain ol' VORP make. Let's take the top ten position players in baseball according to VORP (rounded to the nearest run for reasons having to do with my less-than-brilliant abilities in importing data into MYSQL):

Not too many surprises there. But now, let's compare BP's VORP numbers with their JustVORP numbers for the same players

Not all of those are minor differences. Albert Pujols is almost certainly the best player in professional baseball at the moment, but overrating him by 20 runs -- well, if a team was using VORP to evaluate what they should pay a player, that's around a $10 million difference. Berkman and Holliday's ratings are almost as problematic. Note that all three of those players are from the "easy" end of the defensive spectrum, as well as being from the NL.

I do not want to lose myself in statistical detail that I'm not qualified to go into anyway. The best thing to do would be to go through multiple years and check the standard deviations, the variance, the error, and (as Colin did) the average error. What I will do is easier and more straightforward, if not as definitive. To check out the "positional biases," I went position-by-position (primary position as defined by VORP for players who played multiple positoins), and took all the players with at least 500 plate appearances (400 for catchers and DHs). Then I averaged the difference between VORP and JustVORP by player. Again, this is a measurement of the average over/underrating of each position. It is not the measure of the variance. It is not a multi-year study, and the sample sizes are low. So these only allow us, at best, to from some preliminary hypotheses about VORP's postional biases.

There it is. While the sample size is too small and the method is too crude in our little analysis to say for sure, so far, the hypothesis that VORP overvalues 1Bs and DHs -- the easiest to replace part of the talent pool, and undervalues everyone else --particularly catchers and third basemen -- is borne out. While one or two runs isn't that bad, 7 runs per first basemen is.

(To repeat: we're leaving defense aside for the sake of getting at the offensive and positional adjustment difference. A complete player evaluation must take defense into account, of course.)

Let's take some concrete cases. Ryan Howard, for example, just got a contract extension from the Phillies that will buy out his remaining free agency years. I won't give a full analysis here. Matthew Carruth argues that Howard probably got overpaid. That almost never happens when a player gets bought out pre-free agency, but it did in this case due to the (avoided) arbitration hearings giving too much home runs and RBI and not considering those things that VORP is supposed to look at, such as positional adjustments. But the terms of the signing don't concern us directly here. If you were a GM, and just used VORP from BP (assume defense to be equal), what would your evaluation of Howard's 2008 contribution over a replacement level first baseman be? 36 runs, or about 3.5 wins. Assuming $5 million her marginal win, that implies a salary of $17.5 million a year. JustVORP, on the other hand, says that a Howard was 25 runs above a replacement first baseman. That's about a win, or $5 million, difference. That's a lot of money, even to a baseball team. And in this case, I think that VORP is probalby overvaluing Howard for one or two reasons: 1) VORP proceeds from positional averages, not accounting for the fact that first base is more easily replacable, even with players who normally player other positions, because it is the easiest position to play on the field; and 2) VORP does not adjust for the easier pitching in the National League.

A very different case is Akinori Iwamura, the second baseman of the Tampa Bay Rays. VORP values him at 17 runs above replacement, while JustVORP has him at 27 RAR. Again, that's about a win, or $5 million. The problem again is probably  positional adjustment and league difficulty adjustment, and proportionally, for player who clearly isn't a star like Iwamura, that's big difference -- the difference (assuming average defense) between a 1.7 WAR, below average, stopgap starter who might need to have a replacement available coming up through the minors, and a 2.7 WAR slightly above average player (average being around 2 WAR) who is easily holding down his spot.

The point is clear. Tango, Wyers, and others have pointed out that the runestimator at the heart of VORP is flawed in its treatment of walks and also doubles. When considering positional adjustments and the difference in difficulty between leagues, even in the few cases we looked at here (the top ten players by VORP, as well as Howard and Iwamura) as well as the general "error" by position, it's clear that VORP misvalues more than a few players, and by more than a little. Sabermetrics is supposed to help teams get better evaluations of players. If the assumptions on which JustVORP (and similar "überstats" such as FanGraphs' WAR -- that comparison will be for another column) are true, then in comparison, VORP fails on that score.

Oh, and if you're a sabermetrically-oriented blogger, and don't want to become an out-of-touch old man, take a hint from Murray Chass and Jon Heyman. Stop using VORP. Don't worry, though. BaseRuns and wOBA will bug them even more.

[Brief Update: I have written a supplementary post comparing the VORP and JustVORP numbers for the 2008 Royals over at Royals Review. I don't want to overdo the Royals-related content here, but if you want another concrete comparison of how big the difference can be, there you have it...]