The honest view from inside technical leadership
Analysis

Leadership

Field notes on how engineering leaders keep control of systems as AI changes how software gets built.

Leadership · Response

Developer Friction Metrics Miss the Failures Engineers Learn to Ignore

Google found 84% of pass-to-fail test transitions involved a flaky test. Friction metrics that count lost minutes miss engineers who learn to ignore failures.

Line-art illustration of a CI pipeline panel with two passing checks and one failing cross beside a rerun arrow, representing developer friction metrics that miss failures engineers learn to ignore

Developer friction metrics that count lost minutes miss the most expensive effect of recurring friction: engineers learn to stop believing their own failure signals. Sandeep Khandelwal makes the case for measuring friction in a recent article, and this piece extends one part of it, his formula for the cost of friction, by adding the measure it cannot see.

Khandelwal argues that friction belongs next to output metrics

Khandelwal writes that productivity dashboards track commits, pull requests and deployment frequency, and leave out the time developers lose to slow pipelines, flaky tests, poor error messages, stale documentation and waiting on other teams. He proposes a simple cost: how often an issue occurs, times the time lost each occurrence, times the number of developers affected. In his example, a pipeline failure twice a week that costs 20 developers 25 minutes each adds up to about 16.7 hours a week.

He pairs this with what he calls supportability engineering: systems that explain their own failures through actionable diagnostics, useful error messages, self-service fixes, documented known issues and automated recovery. He also argues that AI-assisted coding moves the bottleneck downstream when the pipeline stays slow. His closing line puts the goal plainly: "It is to stop making them wait."

Google found that most new test failures involved a flaky test

The best public numbers on test flakiness come from Google. In Flaky Tests at Google and How We Mitigate Them, a 2016 post on the Google Testing Blog, John Micco reported that about 1.5% of all test runs gave a flaky result, meaning the same code both passed and failed. Almost 16% of tests had some level of flakiness. He wrote that the rate of new flakiness was about the same as the rate of fixes, so the problem did not shrink with effort.

The figure that matters most for measurement is a different one. Micco reported that about 84% of the transitions from pass to fail that Google’s CI system observed involved a flaky test. He also described the human response. Developers sometimes dismissed a failing result as flaky and later found it was a real failure caused by their code, and he wrote that it is human nature to ignore alarms when a system has a history of false signals.

A developer friction cost counted in minutes improves as engineers stop investigating

Apply Khandelwal’s formula to that behavior. In the first months of a flaky pipeline, engineers investigate each red build, and the time term in the formula is large. Once they learn that most red builds are noise, they press rerun and move on. The time lost per occurrence falls from 25 minutes to perhaps two, and the friction cost on the dashboard drops sharply. Nothing has been fixed. The team has adapted by discounting the signal.

This is the failure mode I want leaders to see. A cost measured in minutes rewards the adaptation that makes the system riskier. The minutes are real, and Khandelwal is right to count them. The larger cost arrives later, when a real regression lands in a test everyone has learned to rerun, and it reaches production with a green build on its second try.

Google’s SRE practice already treats noisy signals as the real cost

Operations teams worked this out for pagers years ago. In the Monitoring Distributed Systems chapter of Google’s Site Reliability Engineering book, written by Rob Ewaschuk, the guidance is that every page should be actionable. The chapter warns that when pages occur too often, employees second-guess, skim or ignore alerts, sometimes missing a real page masked by the noise, and outages run longer as a result. It asks of every alerting rule whether anyone will ever be able to ignore it, knowing it is benign.

A red build is a page sent to a developer. The same test applies. If a failure can routinely be ignored, the signal has already lost its value, whatever its cost in minutes. SRE teams measure alert quality directly for this reason. Developer friction measurement should borrow the practice.

Three measures show whether engineers still trust their failure signals

I propose three numbers to track alongside Khandelwal’s friction cost. First, the rerun before read rate: the share of failed builds rerun before anyone opens the logs. A rising rate means the signal is being discounted. Second, the dismissed then real count: failures marked as flaky that were later linked to a genuine defect or incident. This is the number Micco’s post describes in words, and few teams count it. Third, the false failure rate per pipeline stage, so leaders know which stage is teaching engineers to ignore it.

These measures also cover a gap in automated recovery, which Khandelwal lists as part of supportability engineering. An automatic retry that turns red to green records no lost minutes at all. Unless the retry is logged as an event, the flakiness disappears from every report while it keeps eroding the signal underneath.

Trust measures fail when the labels are dishonest

These numbers have limits. The dismissed then real count depends on someone connecting an incident back to an ignored failure, which takes discipline that many postmortem processes lack. A small team with a fast, stable pipeline may never see enough failures for the rates to mean anything, and the minutes formula will serve it well. Rerun rates can also be pushed down by policy without any change in trust, if engineers are told to investigate first and then do so in name only. As Khandelwal writes of his own metrics, these should drive changes to systems and should never be used to rate individual engineers.

Why This Matters for Engineering Leadership

Khandelwal’s article gives leaders a way to price the time their environment wastes. The Google evidence adds a warning: the worst friction teaches people to stop paying attention, and a time-based cost hides that. From an AI governance view, a failure signal that humans are expected to act on is a control, and a control that is routinely overridden has stopped working. This matters more as teams adopt the AI-assisted diagnostics Khandelwal describes, since he notes they must rest on solid diagnostic data with humans overseeing high-stakes recovery decisions. That oversight is only as good as the signals the humans believe. A leader can ask one question each quarter: of the failures we dismissed as noise, how many turned out to be real?

FAQ

What is developer friction?

Sandeep Khandelwal defines it as wasted effort, delay and cognitive load caused by inefficiencies in how software is built, tested, deployed and maintained. Slow pipelines, flaky tests and poor error messages are common sources.

How do you calculate the cost of developer friction?

Khandelwal’s formula multiplies how often an issue occurs by the time lost each time and the number of developers affected. A failure twice a week that costs 20 developers 25 minutes each comes to about 16.7 hours a week.

How common are flaky tests?

In a 2016 Google Testing Blog post, John Micco reported that about 1.5% of all test runs at Google gave a flaky result and almost 16% of tests had some level of flakiness.

Why do minutes-lost metrics miss the cost of flaky tests?

When engineers stop trusting a red build, they rerun it instead of investigating. The minutes lost fall, so the metric improves, while real failures start to get dismissed as noise.

What should engineering leaders measure alongside lost time?

Measure whether failure signals are believed: how often a failure is rerun before anyone reads it, how often a failure dismissed as flaky later proves real, and the false failure rate of each pipeline stage.

Source: Stop measuring developer productivity by Sandeep Khandelwal, LeadDev, 9 November 2026.

Read more news & analysis