The honest view from inside technical leadership
Analysis

Leadership

Field notes on how engineering leaders keep control of systems as AI changes how software gets built.

Leadership · Response

AI-Assisted Development Needs Kill Criteria for Every Feature It Ships

Microsoft found only about a third of tested ideas improved their target metric. As AI makes features cheap to build, set kill criteria for each before launch.

Line-art illustration of three feature cards on a board, one crossed out beside a pair of scissors, representing kill criteria in AI-assisted development

AI-assisted development makes a feature cheap to build, so the expensive mistake becomes keeping a feature that does not work. Teams should write kill criteria for each feature before they build it. Alan Sharp makes the case for Agile values in a recent article, and this piece extends one claim from it: that knowing what not to build matters more when building is fast.

Sharp argues that AI moved the bottleneck out of coding

Sharp writes that the stages of the software development life cycle are unchanged, while the effort has shifted from development to requirements, testing and customer work. On his teams, tasks that took weeks can reach code review in a day. He argues that the Agile Manifesto says nothing about sprints, estimation or writing code, and that its values and principles fit AI-assisted development well.

Among the 12 principles, he singles out simplicity, defined in the Manifesto as maximizing the amount of work not done. In his words, "knowing what not to build is just as important." He adds that features shipped fast without product-market fit carry maintenance overhead, and that teams can fall into the sunk cost fallacy by finishing work after priorities have changed.

Most features miss their own goal, according to Microsoft’s experiment data

The best public evidence on how often product ideas work comes from controlled experiments. In Online Experimentation at Microsoft, a 2009 paper by Ronny Kohavi, Thomas Crook, Roger Longbotham and colleagues, the authors report that only about a third of ideas tested at Microsoft improved the metrics they were designed to improve. Their estimate for a team that launches ten ideas without measuring them is roughly a third good, a third flat and a third negative. They add that at Amazon, where evaluating every new feature was common practice, the success rate was below 50%.

The authors note a bias in their own number: teams tend to run experiments when they are less sure of an idea. They also argue the bias is smaller than people expect. I treat the one-third figure as an order of magnitude for products that measure carefully. Other teams will see their own rate.

Cheaper building multiplies the failed features a team ships

Put Sharp’s observation next to that figure. If a team’s success rate stays near one in three and AI-assisted development lets it ship twice as many features, it also ships twice as many features that are flat or harmful. Nothing in the Microsoft data says AI changes the hit rate, because the paper predates these tools. My inference is that faster building raises the volume of misses unless something else improves the quality of the ideas going in.

Each miss stays in the product until someone removes it. It needs tests, dependency upgrades, documentation and support. This is the maintenance overhead Sharp describes, and it grows with every flat feature that nobody turns off.

Launched features rarely come out without a rule written in advance

Kohavi and colleagues are blunt about removal. They write that once a team launches an idea instead of testing it, backing it out is expensive. They give an example from Office Online, which changed its rating system from yes or no to five stars and lost over 80% of responses. The paper says it took eight months to detect, analyze and replace that version. They also write that if a metric drops by 3%, the chance anyone starts a project to back out a feature they launched is minuscule.

Sharp applies the sunk cost fallacy to work in progress. The same pull applies after launch, with more force, because a shipped feature has a team that is proud of it. The countermeasure is to decide the removal rule before any code exists. I propose four fields for each feature brief: the metric it should move, the threshold that counts as success, the date of the review, and the person who will remove it if it misses. When AI makes building cheap, this brief may take longer to write than the first working version takes to build. That is the point of it.

Kill criteria fail where effects cannot be measured

The rule has limits. Kohavi and colleagues write that for most ideas the change in key metrics is too small to detect over time without a controlled experiment. A low-traffic internal tool or an early-stage product may never gather enough data to reach a verdict, and a threshold it cannot measure becomes a ritual. Security fixes, compliance work and infrastructure upgrades often have no usage metric at all, and removing them on usage grounds would be a mistake. Those features need a review rule based on risk or obligation instead.

There is also a quieter failure. Teams can set thresholds low enough that everything passes. The control for that is to record who set the criteria and to review the share of features that were actually removed. If the answer is zero over a year, the criteria are not doing any work.

Why This Matters for Engineering Leadership

Sharp’s article argues that AI-assisted development moves the pace of a team away from code. The Microsoft data adds a harder number: most ideas do not work, and shipped features tend to stay. From an AI governance view, a kill decision is a decision right, and it needs an owner in the same way a production deployment does. Leaders who adopt AI coding tools can ask one measurable question each quarter: how many features did we remove because they missed criteria we wrote down in advance? A team that ships faster and never removes anything is accumulating maintenance it chose by default.

FAQ

What are kill criteria for a feature?

They are a metric, a threshold, a review date and a named owner, written before a feature is built. If the feature misses the threshold by the date, the owner removes it.

Why does AI-assisted development make kill criteria more important?

AI lowers the cost of building a feature. If the share of ideas that work stays the same, a team that ships more features also ships more features that do not work.

How many product ideas actually improve their target metric?

In a 2009 paper, Ronny Kohavi and colleagues reported that only about a third of ideas tested in controlled experiments at Microsoft improved the metrics they were designed to improve.

Why do teams keep features that do not work?

Kohavi and colleagues wrote that once a feature is launched, backing it out is expensive, and a small metric drop is unlikely to start a removal project. Alan Sharp’s article names the sunk cost fallacy as a reason teams keep finishing work after priorities change.

Do kill criteria work for every feature?

No. Low-traffic products may not be able to measure small effects, and security, compliance and infrastructure work often has no usage metric. Those features need a different review rule.

Source: Agile matters more in the AI era by Alan Sharp, LeadDev, 4 November 2026.

Read more news & analysis