A proof of concept that passes on a sample shows that a solution can work, and it cannot show that the solution works on every record. Manas Verma makes this point in a recent article about forward deployed engineering, and this piece extends it with a way to put a number on the gap.
Verma says a sample cannot prove a solution works on messy real data
Verma is an AI forward deployed engineer, and his article describes what the job looks like once the demo is over. He lists work that has little to do with building agents: a client data migration with no data dictionary, for which he built a partial one; a large data load to a client’s staging environment that kept hitting resource limits; and AI assistants that invented column names that did not exist and missed required columns in a table. His conclusion is that the role is moving from AI demos toward reliable production systems with more deterministic outcomes.
His central point about proofs of concept is that a solution tested on a sample of records cannot show that it works for every record in messy real data. He also says customers do not always know what they need, so requirements often have to be validated, reinterpreted, or solved differently, and that AI tools help with retrieving context but need careful human review.
A clean sample still leaves a failure rate you can calculate
Statistics already has a name for the gap Verma describes. If an event occurs zero times in N independent trials, the upper limit of the 95 percent confidence interval for its frequency is approximately 3/N. A 1995 letter in JAMA by Thomas Newman restates this rule, which originates with Hanley and Lippman-Hand, and gives two examples: 50 patients with no events leave a limit of about 6 percent, and 1,000 patients leave about 0.3 percent. The Wikipedia entry on the rule of three lists its assumptions: independent trials with a constant probability, a rate close to zero, and a sample size above about 30.
Apply that to a pilot. A team tests an AI extraction step on 50 records, all 50 come out right, and the demo is declared a success. The data supports a claim that the true failure rate is below roughly 6 percent. On a hypothetical migration of 2 million records, 6 percent allows for up to 120,000 bad records. Nobody in the room said that, and the demo implied the opposite.
Agree a failure bound before the pilot starts
The practical step is to set the acceptable failure rate before anyone runs the pilot. I call it a failure bound, and the name is mine. Rearranging the same rule gives the sample size: to support a bound of 0.5 percent with zero failures, a team needs about 600 clean, independently drawn records, since 3 divided by 600 is 0.005. A bound of 6 percent needs only 50. Because the bound comes first, the pilot size is derived from the risk and not from what was convenient to test.
Three conditions make the number honest. The records must be drawn at random from the full population, and a hand-picked sample breaks the independence the rule assumes. Known messy classes, such as legacy tables with no data dictionary, should each get their own draw, so that the clean records from one class do not vouch for another. Any failure in the sample ends the claim, and the team investigates the cause before drawing again. For the full run, the pilot result is replaced by reconciliation: every record in must be accounted for as either a record out or a rejected record with a stated reason.
The rule fails on correlated errors and on cheap full checks
The boundary conditions matter. The rule of three assumes independent trials. Messy data fails in clusters, for example when one legacy source system wrote every date in a different format, so 600 clean records from the other systems say nothing about that cluster. Stratified draws reduce this problem, and they cannot remove it, because nobody knows every cluster in advance. The rule also treats all failures as equal, while a wrong column name in a log table and a wrong column in a billing table do not cost the same. And where a deterministic validator can check every record cheaply, a full scan beats any sample. Sampling matters most where a human has to review each output, which is the situation Verma describes with AI assistants that need careful review.
There is a fair case for the demo as well. A small pilot is a good way to learn requirements, and Verma’s own examples show customers discovering what they need through contact with a working system. The failure bound does not remove that value. It stops the learning exercise from being reported as evidence of reliability.
The U.S. National Institute of Standards and Technology asks for something similar in its AI Risk Management Framework 1.0. The framework says accuracy measurements should be paired with clearly defined and realistic test sets that are representative of conditions of expected use. Its MEASURE guidance calls for documenting test sets and metrics, demonstrating validity for conditions similar to the deployment setting, and documenting the limits on generalizing beyond the tested conditions. The framework also warns that laboratory measurements may differ from risks that emerge in operational, real-world settings. A failure bound is one concrete way to record that limit in a sentence a sponsor can read.
Why This Matters for Engineering Leadership
Leaders approve production rollouts on the strength of pilots, and a pilot summary that says “it worked on our test set” hides the quantity they most need, which is how bad the untested remainder could be. Asking for the failure bound before the pilot begins gives leaders a decision rule agreed in advance, in the same spirit as a release gate. From an AI governance view, it also creates a record of what was claimed, on what evidence, and with what stated limit, so accountability can be traced if the full run fails. Engineers closest to the customer, such as the forward deployed engineers Verma writes about, are well placed to propose the bound, since they know which data classes are messy.
FAQ
Why does a successful AI pilot on a sample not prove the solution works?
A sample shows the solution can work on those records. With zero failures in N independent trials, the 95 percent upper limit on the failure rate is still about 3/N, so a clean 50-record pilot leaves a limit near 6 percent.
What is the rule of three?
If an event occurs zero times in N independent trials, the upper limit of the 95 percent confidence interval for its frequency is approximately 3/N. Thomas Newman restated it in a 1995 JAMA letter, crediting Hanley and Lippman-Hand.
What is a failure bound?
It is the highest failure rate a team accepts, agreed before the pilot starts. The term is this piece’s proposal. The bound sets how many clean, randomly drawn records the pilot needs, for example about 600 for a 0.5 percent bound.
When does the rule of three not apply?
It assumes independent trials with a constant failure probability. Clustered errors, such as one legacy source system with its own date format, break that assumption, and a hand-picked sample breaks it too.
What does NIST say about test sets for AI systems?
NIST AI RMF 1.0 says accuracy measurements should be paired with clearly defined and realistic test sets that are representative of conditions of expected use, and that the limits of generalizing beyond tested conditions should be documented.
What did Manas Verma say about proofs of concept?
He said a solution tested on a sample cannot show that it works for every record in messy real data, and that forward deployed engineering is moving from AI demos toward reliable production systems.