Last Friday, MLB Network completed their Top Ten Right Now series with a look at baseball’s best ten managers. The series was informed heavily by Clubhouse Confidential’s “shredder” which uses an objective formula to determine player value much like Wins Above Replacement. But this gets sticky, as they admit, when it comes to managers because it’s very hard to determine what managers should get credit for and for what they should get blamed.

So I started thinking about what it would look like to quantify manager value. We have a great set of offensive and pitching statistics that quantify player performance and an increasingly good group of stats to measure defense, but what do we have that measures what a manager does? Let’s first ask the question, what is it that managers do?

Managers Lead and Motivate Men

This is definitely the hardest piece to measure. How do we quantify that which we cannot observe? And even if we did observe it by being inside meetings and next to the skipper in the dugout, how much does this leadership matter?

It’s conventional wisdom that this aspect of the job matters a great deal and that players often improve and weaken their performance based on how well a manager motivates and leads. A good number of people in the analytic community scoff at this notion because world class, competitive athletes are already so far to the right on the distribution of talent that this type of leadership shouldn’t be terribly effective at making a noticeable difference in performance.

I tend to see a big role for leadership and motivation because this also includes communicating with players about their roles and what is expected of them. In general, this aspect of managing is important, but very hard to measure. Let’s leave this aside except to say that managers who are good leaders will get outcomes no worse than those who are bad leaders.

Managers Make Personnel Decisions

This should be much easier. Putting the right people in the right spot in the lineup and calling on the right players to pitch at the right time is a very important aspect of the job. Hitting your best hitter in the right spot is a better choice that hitting them ninth. Most in the statistical community would argue for putting your best hitter second, but we’ll setting for third until we win that war. You can also add value here by platooning the right players and giving players appropriate days off too.

On the mound, knowing when to pull your starter and knowing which reliever to use when is perhaps the place a manager can do the most good or harm.

I’d be curious to see how much an optimal lineup could improve a team over worst case lineup. But on the other hand, no manager would actually use the worst possible lineup. How many wins can be gained between optimization and ordinary? That’s the question we have to answer.

The pitching side would be tricky, but it is possible in theory. What we need is to gather leverage indexes of every plate appearances and then somehow compare those indexes to the outcomes in cases when the manager makes a move and when they don’t. This could be problematic, because managers can only make a move if they have the right personnel on the roster, but we could find a way to compare these moves to each based on the roster they have. I don’t have a formula in mind, but I think evaluating each pitching change and non-pitching change and the outcome on the next series of at bats would be the proper way to look at this if we created some sort of constant that accounted for the best a manager could do with the team he has.

Neither of these ideas are fully formed, but it should help us understand managers just by thinking about what it would look like to measure them.

In-Game Strategy

Strip out the personnel choices and managers have moves to make like calling bunts, hits-and-runs, and stolen bases. Two problems emerge. The first is obvious and unavoidable, sometimes players do these things on their own. Can’t do anything about that. Let’s move on.

The second issue is game theory. How is it that we can judge a manager based on incomplete information? If a manager thinks a stolen base is a good idea, he is weighing the likelihood of success given what he anticipates the other manager will do (pitch out, pickoff, nothing). The problem here is that if Manager A calls for a steal and it works, it might work in spite of the fact that he made a bad choice. Manager B may have called for a pitchout, but the catcher may have dropped the pitch. In this case, Manager A made the wrong call, but got the right result. How do we account for that? It’s tough.

You’re starting to see why this is so hard.

Putting it Together

So managers are leaders, make personnel choices, and make in-game strategic decisions. The first hard to measure. The second is easier, but requires a lot of complicated math. The third is pretty easy, but requires us to control for what the other manager does, which is theoretically quite problematic.

But just in thinking about these aspects, it gives us a more formal understanding of what we are looking for and allows us to better evaluate managers using the tools we do have.

Like our eyes.

I know that’s a really lame answer, but it’s a good one. Two issues emerge. The first is that I can’t watch every manager enough. I can watch my team’s manager very closely and judge him well, but I can’t watch all thirty. The second issue is objectivity. Your eyes aren’t as objective as a more math-y approach, but a math-y approach isn’t without bias.

The best way to evaluate managers, for now at least, is to think very formally about what managers do and then judge them with your eyes. We want to see managers bringing in relievers at the right times. This requires us to decide before every hitter whether or not we think he should go to the pen. We can’t judge him by saying, “I knew it!” We have to say, “he should pull his guy now,” and then see how often we are right and how often he is. We have to manage alongside him, not in retrospect.

We also need to understand strategy better by looking at Run Expectancy Matrices and considering how likely a given outcome will be and how often the managers play the high probability moves.

None of this is easy, and it is very subjective, but as my Game Theory professor says, formalizing allows us to isolate that which we are debating and allows us to have specific conversations about the items of interest to us. In English, when we decide what we are looking for before we go looking for it, we can measure our expectations against the outcomes to see if managers are getting the most out of their players.

While we probably won’t ever have a WAR for managers, this discussion will hopefully help you better judge your team’s manager. Try it out and let me know what you think. For reference, I’d say Jim Leyland is top notch at leading men, pretty good at offensive personnel, good at deciding when to pull the starter, shaky when it comes to using his bullpen, and average when it comes to in-game strategy.

Tell us what you think about grading managers by typing it on your “Hand Computers.” Bonus Points if you know from where that reference originates.