How Much Does Coaching Matter?

My last post where I attempted to model the upcoming season resulted in two posters (Tony and anonymous) asking about the effect of new head coach John Anderson. This is a great question, but also one that is hard to answer so let me take a crack at it.

Coaching is one of the great mysteries of performance analysis. Let me start by talking about baseball where the statistics allow us to break down the game much more precisely. Most baseball analysts find little evidence that managers make a big difference. Their conclusions is that if coaching matters it tends to show up a) bullpen usage b) strategic choices (bunting, pinch hitting, etc) in close ball games.

For a real world example consider the case of famed Braves pitching coach Leo Mazzone who moved to Baltimore. A number of baseball analysts were curious to see if he the Baltimore staff would show evidence of "unexpected" improvement with Mazzone's arrival, but that turned out not to be the case. There was no dramatic improvement in Baltimore's pitching beyond what you might expect based on their talent level and margin of error.

What about hockey? You could make the case that in team sports like football and hockey coaching matters more than in baseball where it is a one-on-one confrontation between a single batter and a single pitcher. In hockey and football the guys may play well together as a unit to achieve success.

My own opinion is that coaching in hockey probably most strongly manifests itself in special teams. Why? Because pre-designed "plays" are most likely to show up in PP and SH situations on a regular basis. When you're on the PP you can count on maintaining possession of the puck in the offensive zone for an extended period of time. In contrast, at even strength it is much more difficult to keep puck possession and run specific strategies or plays--yes you can do it, but player raw skill and improvisation abilities also matter a great deal at ES.

If you look around the NHL every year you see some skilled teams that struggle with special teams and you also see weak teams with great special teams. That suggests to me that more than raw player skill level is at work here and these oddities are caused by either coaching or luck.

You also have to consider coaching within a competitive environment. Certainly John Anderson will be doing everything he can to get the most out of his roster--but the same thing can be said for every opposition coach. So when you're attempting to forecast all the Eastern conference teams I have a hard time giving the Thrashers any extra points over say Tampa who will also have a new coach as well as new players.

Is there any empirical evidence that coaching makes a big difference? Perhaps. Take a look at the list of winners of the Jack Adams Award (Coach of the Year) and you will see that that vast majority of them are coaches who took over bad teams and the team showed a big improvement. So there is some anecdotal evidence of a "new coach bump" out there in the data.

But if you look hard at some of these "Coach of the Year" cases what you see at times is the new coach getting credit for things he might not deserve to get credit for. Let me try and explain. In some cases the "new coach" bump is just random luck--for example if a team was riddled with injuries the season before and they return to being healthy the following season--that team should be better--but almost invariable the hockey media is going to credit that improvement to the new coach when in fact it is health (a luck factor) at work. Likewise players have a certain skill level but they also have variation around that fundamental skill level. If a team had several players with down years (i.e. Kozlov, Exelby, Havelid) and those players return to their previous level of performance the coach usually gets the credit--when it is could be that the player had a hidden injury or off-ice distraction that depressed their performance the previous season. Again the coach will usually be credited for this rebound. In many cases when people say "coaching" produced this, they simply capturing luck or rebound performances with the coaching variable.

Does every coach produce a "new coach bump"? No. Some new bench bosses see their teams struggle and get fired after just one year. If you're a new head coach taking over a club without any major injuries or down years by key players the year before, look out because you may be on the short road to a pink slip.

So if you're trying to forecast how do you handle this? Should I assume that both John Anderson and Barry Melrose will get that "new coach" kick, or will it just be Anderson? Because I don't have any feel for this right now I'm just assuming that it will be within my margin of error for the forecast.

So that's a long explanation but until I see empirical evidence to the contrary I tend to think that most of the "new coach bump" can be at the player level--you project an injured guy to be healthy or a slumping guy to rebound some. Special teams are the key area where Anderson might be able to squeeze out some extra points not predicted by the team talent level.

Let me conclude by saying this. I'll be more than happy to write a post at the end of the 2008-09 season titled "why my model/prediction was wrong" if that is the case. But last year my model had us around 10th in the conference and we finished 14th (mostly because Zhitnik was much worse than I expected). If we make the playoffs I'll be more than happy to say I was wrong, but right now I'm not sweating it.