With Alex Avila out for an extended period of time, James McCann stepped in to take the everyday catcher reins. Offensively, he’s provided a solid .264/.293/.400 line (90 wRC+), which puts him slightly above the average catcher at the plate. He’s not Buster Posey, and won’t ever be, but he’s an above average hitter in his first season at a position that usually lags behind in offensive development.

We’ve also seen McCann nail plenty of would-be-base-stealers. According to Baseball-Prospectus’ new base running statistic, James McCann has been the 7th best catcher when it comes to preventing steals per opportunity. The names ahead of him this year are: Molina, Posey, the other McCann, Bethancourt, Ramos, and Realmuto. For the most part, this is a who’s who of base runner assassins.

His bat checks out and his arm is doing some heavy lifting. The only thing left for McCann to do is receive the pitching staff. And that’s where we run into some concerns.  By the most popular metrics, McCann has been a terrible pitch framer. BP has him 3rd worst at -21 strikes (about -3 runs) and Stat Corner has him at -50 calls and about -7 runs.

Before we get into the real analysis, a word of explanation. First, the two numbers are different because BP uses a more sophisticated model that attempts to control for a lot of other factors. StatCorner is basically just modeling the probability of a strike and whether he gets it. As a result, you would expect to see the BP numbers more constrained. In other words, pay attention to the fact that both models have him in the bottom three, and that they say he’s costing the team 3-7 runs with his glove so far. You can probably expand that out to 1-2 wins per season if it were to continue.

So if this is true, it’s a problem. Avila’s framing has been up and down over the years, but has seemingly been a very effective backstop. How do we make sense of this new data with McCann?

Sample Size

Before we go any further, McCann is only a couple thousand framing chances into his big league career. While the people who make the framing stats will tell you that a couple thousand pitches is a pretty good sample, I don’t agree with that perspective entirely. For one, there may be a learning curve for catchers making their debuts for lots of reasons. Learning the pitcher, paying attention to their offense, not knowing the umpires well. 2000 pitches might be a fine sample for a veteran, but I’m not sold that can be applied to rookies.

Second, we don’t actually care much about the total number of pitches eligible for framing. Obviously, we want to exclude swings and the stats do that, but not every pitch is equally framable. This is also factored into the metrics, as they hang a probability of being called a strike into the model. But let’s say we’re looking at 100 called pitches. Then let’s say you wind up with +2 strikes for that game. That’s like .3 runs or so of value above average.

The key, however, for determining ability is how well you did on each individual pitch. If you have 100 pitches but 95 of them were going to be a ball or strike 95% of the time, you are only being judged on five pitches, essentially. Where the pitcher throws the ball not only determines how valuable a frame job is, but how many chances you have to show your stuff. And we don’t know how those chances are distributed per catcher each year.

For this reason, the high level data does not convince me that McCann is a bad framer. It doesn’t prove the opposite, but I want to be clear that the observations we have so far could be misleading.

Framing is hard to measure

While I admire the work that’s been done on framing, it’s far from finished. I won’t go into a ton of detail, but while framing stats control for lots of factors, the models treat the observations as independent when they most certainly are not. Imagine a pitch perfectly on the corner gets called a ball even though it was well-framed. Based on every factor involved, the umpire would normally call that a strike 75% of the time (number made up). But he calls it a ball.

If that happens, the model updates its prediction about the umpire’s zone, and let’s say the identical pitch would now be a predicted strike 74% of the time. Let’s pretend the exact same pitch unfolds right after. Same everything.

Ball or strike? The odds still say strike, but the umpire is probably more likely to call a ball because of the path dependent nature of the last call. Umpires have variable strike zones and they make a clear effort to maintain a single zone over the course of  game. If they blow one call, they likely feel obligated to maintain that zone for the rest of the game, at least.

Now I don’t have numbers to demonstrate this theory, but I believe very strongly that it is true. And if it’s true, the framing metrics we have are much less precise than we think they are. Without getting off on a tangent about where I’d like to see the research go, I’ll leave you with this point as it relates to McCann. Experienced catchers are in a better position to adapt to these changing conditions. McCann might have framed both pitches beautifully, but he’s not getting credit. Yadier Molina might learn from the first and adjust for the second. It comes with experience, not ability. Framing is partly mental and you would assume the mental aspect creates a larger snowball effect.

Basically, new catchers are more open to deviations from true talent. And that could be happening with McCann. I don’t know if it is, I’m just saying the numbers on the page could easily be misleading about how well he’s framed pitches.

Because that’s just it. Framing is about making a ball look more like a strike and about making a strike look like a strike. If you were to contend that McCann has gotten fewer calls than the average catcher so far, it would be harder to argue than if you said he hasn’t framed well. A 75% strike and a 60% strike have varying degrees of difficulty, but if you get five 60% pitches and one 75%, a “26% framer” only gets one call even if he does the same job on each pitch. In other words, the payoff is ‘yes/no’ while the skill is not. That’s a problem.

So let’s talk about McCann

All of this prelude has suggested that there are lots of non-McCann reasons why the numbers themselves shouldn’t worry you. I don’t really think the first 2000 frames are reflective of a catcher’s talent level. There’s a lot going on about framing that we haven’t measured just right. The real thing we want to measure is where the ball crosses the plate relative to where the umpire thinks it crosses the plate. It doesn’t matter if you get an extra strike, it matters how much you increase the probability of a strike. The metrics measure you in the aggregate and that might not tell us what we want to know.

Baseball Savant has 2,745 framable pitches for McCann. Let’s compare balls and strikes.

Now let’s try Posey, who rates very well.

The different sample sizes are hard to parse, but notice in McCann’s there are a few more balls called in the center of the zone. This is a great example of the problem we might want to study. The umpire might have totally blown those calls, but McCann gets dinged because the umpire almost always gets it right. If the umpire gets the call right 98% of the time, but is 100% to blame for the mistake in this case, the catcher gets a huge penalty even though it isn’t his fault.

What we really want to do is evaluate McCann’s performance on individual pitches to see if he was responsible for the calls that went for and against him. Here’s the worst example of this problem. Not his job to frame this pitch and the umpire gives up on it.

Granted, I’m not showing you all of the pitches because that would be crazy. That’s what we want to watch going forward. Look for the calls McCann doesn’t get and decide if he did something wrong. We’ll revisit this when there’s more data.