Engineering Manager Ratings: Why Calibration Needs an Evidence File

Engineering manager ratings are less consistent than engineer ratings, so a manager needs a stronger written record of their own decisions before a calibration meeting starts. Stephane Moreau makes this case in a recent article, and this piece extends one claim from it.

Moreau says a manager’s evidence is other people’s output

Moreau argues that engineering managers are harder to evaluate than engineers because their results come through their teams. "Your team’s achievements don’t automatically show your contribution." A manager can lead a strong team and still be unable to show which outcomes came from their own decisions. He adds that a manager’s case is argued in calibration by someone else, a manager’s own manager, in front of peers who have not seen the work.

He cites a large study of performance ratings in which supervisors agreed less when rating managers than when rating individual contributors, and he suggests evidence inputs such as team sentiment, 360 feedback, engagement scores, and skip-level conversations. His advice is to tie outcomes to specific decisions, to prepare evidence the manager’s own manager can use in the room, and to take on problems that cross team boundaries once the team is performing well.

Rater agreement of .57 means manager ratings need more documentation than engineer ratings

The study Moreau points to is an updated meta-analysis of the interrater reliability of supervisory performance ratings by You Zhou, Paul Sackett, Winny Shen, and Adam Beatty, published in the Journal of Applied Psychology in 2024. Across 132 independent samples, it reports interrater reliability of .57 for managerial positions and .68 for nonmanagerial positions, with an overall estimate of .65. Read the usual way, a coefficient of .57 leaves 43 percent of the variation in ratings as something the two raters do not share.

An earlier benchmark is lower. Viswesvaran, Ones, and Schmidt reported in the Journal of Applied Psychology in 1996 that the mean interrater reliability of supervisory ratings of overall job performance was .52. The 2024 authors advise against relying on a single grand mean and recommend job-specific figures. For this argument the point is narrow. In the best current estimate, the managerial figure sits below the nonmanagerial one, so a manager has less margin for a rating that rests on impression alone.

A one-page decision record gives calibration the same evidence for every manager

Moreau tells managers to prepare evidence. The gap I see is that nothing tells a calibration room what shape the evidence should take, so each manager’s case arrives in a different format and the best storyteller has the advantage. A shared format would reduce that advantage. I propose a one-page decision record for each material outcome, with the same five fields every time: the decision the manager made, the date, the alternatives that were considered, the measure that moved afterward, and the people outside the team who can confirm it. The name and the five fields are my own proposal, and I have not seen a study that tests it.

This also follows an AI governance habit. Accountability for any decision depends on a record of who decided what and why. A manager whose decisions are written down can be assessed on what they decided. A manager whose record exists only in memory is assessed on what the room remembers. The same record helps the calibration lead, who has to speak for a manager who is not present, and it gives cross-team impact, which Moreau says is easier to recognize, a place to be written down.

Agreement is not accuracy, and a decision record has costs

The studies measure whether two raters agree. They do not show that either rater is correct, so a figure of .57 does not mean manager ratings are wrong. I also have not found a primary study showing whether calibration meetings raise the reliability of manager ratings, so I cannot say how much a decision record would help. The record has real costs. It takes time to write, and a well-written record can still hide a weak decision behind clear prose. It can also push managers to claim credit for outcomes their teams produced. The calibration lead should therefore ask the same questions of every record, including what the team did without the manager’s involvement.

Why This Matters for Engineering Leadership

Leaders who run calibration decide pay, promotion, and retention on evidence that differs between managers and engineers. If managers are rated with less agreement, the process deserves more structure, and the cheapest structure is a common evidence format agreed before the cycle opens. Senior leaders can start by asking each manager for decision records ahead of the meeting, and by asking the room whether two supervisors would have reached the same rating from the same record. Doing so also protects managers whose work is quiet, such as removing a dependency, which is easy to overlook when nobody wrote it down.

FAQ