Training Data Poisoning: Why Passing Tests Proves Little

Training data poisoning can ship a compromised model through every test an engineering team runs, because those tests never look at the data. Upendra Jadon makes this case in a recent article, and this piece extends one claim from it: the owner of a training corpus should be able to list every party who can write to it.

Training data poisoning hides in the gap between tests and data

Jadon argues that training data acts as the logic of an AI system, so corrupting it changes behavior without any code change a reviewer would recognize. He describes four ways attackers manipulate a model: availability attacks that degrade accuracy with noisy samples, integrity attacks that skew one input class while overall performance looks normal, backdoor attacks that plant a trigger in a few samples, and attacks aimed at one demographic group. He calls backdoors the most important for engineering leaders, because the model behaves normally until the trigger appears. He uses healthcare as his example, since clinical AI draws data from many independent sites and each one is a possible injection point.

His point about engineering practice is that CI/CD, unit tests, and security scanners do not evaluate training data, so a poisoned model can clear all of them. His instruction is "Treat training data as production infrastructure." He recommends provenance records for each dataset, threat modeling for training data, adversarial testing before deployment, post-deployment monitoring by input segment, and shared ownership across governance, security, and ML engineering.

A fixed handful of poisoned documents can be enough, whatever the corpus size

Anthropic published a study on 9 October 2025, run with the UK AI Security Institute and The Alan Turing Institute, that bears directly on how much data a leader needs to worry about. The team trained 72 models from 600 million to 13 billion parameters. It found that about 250 poisoned documents reliably planted a backdoor at every size, while 100 were not enough to do so robustly. The 250 documents were roughly 420,000 tokens, about 0.00016% of the training tokens. The authors conclude that poisoning needs a near-constant number of documents regardless of model and training data size, so the absolute count matters more than the share.

The implication for review is plain. Checking a fixed percentage of a large corpus is unlikely to catch a fixed handful of records, and the comfort of "our corpus is huge, so a few bad records are diluted" is not supported by this result. The study has limits that the authors state. The backdoor was a trigger phrase, <SUDO>, that made the model output gibberish, which is narrow and low-stakes. It is unclear whether the pattern holds at larger scales or for harder behaviors such as writing insecure code or bypassing safety guardrails. I would read it as a reason to track who can write to a corpus, and not as proof that every model is at risk.

Backdoors survive validation and travel through pre-trained models

The idea is older than language models. Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg described BadNets in a paper first submitted on 22 August 2017. A BadNet performs well on the user’s training and validation data and misbehaves on inputs the attacker chooses. Their demonstration was a street sign classifier in which a sticker on a stop sign made the model read a speed limit. The backdoor also survived when the network was retrained for another task, cutting accuracy by 25% on average when the trigger was present. The abstract warns that outsourced training introduces new security risks and notes that users often fine-tune pre-trained models.

This adds a boundary to Jadon’s provenance advice. A team that keeps careful records of its own fine-tuning data still inherits whatever is inside the base model it started from. Its register can record the origin and exact version of that base model, but it cannot see the pre-training corpus. That is a gap I do not know how to close from the outside, and leaders should say so in their risk register.

A write-access register turns provenance into a release gate

Jadon recommends a provenance record for each dataset covering its origin, who has write access, and when each entry arrived. A record that nobody checks is documentation. I propose making it a gate, and I call the result a write-access register; the name is mine. Every dataset used for training has an entry listing the people and pipelines that can write to it. A model cannot be promoted if its data contains writes that no register entry explains. Every change to the corpus gets a reviewed diff and a named approver, in the same way code changes do. The register also names the base model and its exact version.

The register has limits. It cannot detect poison that arrives through an authorized source, so a compromised hospital site that is on the list passes the gate, and Jadon’s adversarial testing and segment monitoring remain necessary. It adds delivery time on every data change. For a small dataset curated by one team, the register is a short document and little more. Its value grows with the number of independent writers, which is why Jadon’s healthcare example is the hard case.

Why This Matters for Engineering Leadership

Most leaders can say who approved the last code change to a production service. Few can say who approved the last change to the data a model learned from. From an AI governance view, a model whose training data has no named owner and no list of writers has no accountable owner either. A leader can start with three questions: who can write to the data our models learn from, which change added the newest records, and who approved it. If the answers take days to assemble, the register is the first piece of work.

FAQ