Incident-Free Years Are Not Proof of Safe Engineering

Incident-free years do not show that a system is safe, because an incident count records the failures someone noticed and says nothing about the controls behind them. Dmitry Moskalyuk makes this case in a recent article about a bank’s integration system, and this piece extends that one claim.

Moskalyuk found business logic with no version control, tests or review

Moskalyuk describes a bank system that connected to pension funds, tax authorities, retail partners and government services. Its business logic lived in database rows. Those rows had no version control, no tests and no review. Nobody complained, and the quiet was read as health. He writes that the risks were structural, and that leadership accepted the fix of Git, CI/CD and a test framework only when he framed it as a business requirement.

The cleanup then surfaced three things the team had not seen: a script that had failed silently for two months after an interpreter change, a memory leak the team had known about for years and never fixed, and a large amount of duplicated code. His summary of the lesson is that "Years without incidents were luck, not proof of safety."

A clean incident record measures how well failures are detected

An incident log contains the failures that somebody saw and chose to record. The silent script in Moskalyuk’s story is the clearest example, because for two months it failed in a way that no incident count could contain. If the chance of seeing a failure is low, the record stays clean whether the controls are good or absent. That is why an incident-free year is a weak signal on its own. It tells a leader something about detection and about team care, and much less about whether a change can be made, reviewed and reversed safely.

The practical consequence is that the useful question changes from "how many incidents did we have?" to "if this component failed quietly tomorrow, how would we find out, and how long would it take?"

Feynman and the Challenger record show the same inference error

Richard Feynman made the point about the Space Shuttle in his personal observations in the Rogers Commission report, Appendix F. He wrote about the solid rocket booster O-rings, where earlier flights had shown erosion and the program had treated those flights as acceptable. His summary of the reasoning was that "The acceptance and success of these flights is taken as evidence of safety." He argued that erosion was a warning that something was wrong, and that safety could not be inferred from it, comparing the practice to Russian roulette. A run of flights that came back is a run of outcomes. It is not a finding about the margin.

The bank and the shuttle are very different systems, and I am not equating them. The shared structure is narrower: a record of past success was used to stand in for an understanding of the mechanism.

Knight Capital shows what an unreviewed deployment path can cost

The U.S. Securities and Exchange Commission’s administrative order against Knight Capital gives a documented case of a deployment path with no second check. Knight rolled out new code to the eight servers of its SMARS order router, starting on 27 July 2012. A technician did not copy the new code to one of them, and Knight had no second technician review the deployment and no written procedure requiring one. The order also records that Knight had stopped using a function called Power Peg in 2003 but left the code on the production servers. On 1 August 2012 the one server that missed the update ran the old code. In about 45 minutes the router produced roughly 4 million executions in 154 stocks, totaling more than 397 million shares. Knight lost over $460 million, and the SEC imposed a $12 million civil penalty after finding violations of the market access rule, Rule 15c3-5.

Knight’s technology is not the bank’s, and nothing in Moskalyuk’s article says his bank had the same exposure. The link is the control environment. In both stories the risk sat in how changes reached production and in code nobody reviewed, and a long quiet period preceded the failure.

Version control is necessary and still leaves boundary conditions

There are limits to this argument. Moving logic into version control and CI does not make a system correct. A thin test suite can create false confidence, and a review process that approves everything adds ceremony and little protection. Migration also costs real effort, and Moskalyuk himself reports that the hardest work was social. On a small system with one owner and low consequences, the cost of a full change pipeline may exceed the risk it removes. The case for acting grows with the consequence of failure and with the number of people and partners who depend on the output.

The same reasoning applies to AI systems. A prompt, a routing rule, a policy threshold or an agent permission that lives in a console field or a database row is business logic. It can change without a reviewer, a test or a record of who changed it. Whether a given organization’s AI configuration is managed this way is an empirical question, and leaders can answer it by asking.

Why This Matters for Engineering Leadership

Moskalyuk recommends asking teams where business logic actually lives and how changes reach production. I would add a third question: how would we learn if this job stopped producing correct output? A team that can answer all three has evidence about control. A team that points to a clean incident record has evidence about luck or about weak detection, and the leader cannot tell which without asking. From an AI governance view, accountability depends on being able to say who changed a rule, who reviewed it and how a failure would surface. Regulators and auditors ask for exactly that after an incident, and the SEC’s findings on Knight are an example of what the answer can look like when nobody wrote it down. Leaders who schedule that work on their own timeline keep the choice of when it happens.

FAQ