The honest view from inside technical leadership
Analysis

Engineering leadership

Independent reporting on the decisions, costs and contracts that shape how engineering organizations build with AI.

Security · Response

AI Coding Rules Need the Change Control NIST Already Applies to Compilers

Curated AI coding rules cut measured risk by up to 70% in one vendor benchmark. NIST’s SSDF already says how to govern that: version, review, retest on upgrade.

Line-art illustration of a rules document with a version tag feeding a code generator gear, with a test gate between them, representing change control for AI coding rules

AI coding rules should go through the same change control a team already applies to its compiler settings, because the rules file now decides a large share of how much vulnerable code the team ships. Michael Hill reports in a recent article that curated rules moved measured security risk more than the choice of model, and this piece follows one of his closing points: retest the rules every time the model changes.

Hill reports that AI coding rules moved risk more than model choice

Hill covers the second wave of Secure Code Warrior’s AI Trust Index, released on November 17. The company ran more than 30 models inside a harness of prompts, rules and tools, building more than 15,000 apps. The models produced between 0.24 and 3.66 vulnerabilities per 1,000 lines of code. Vague instructions such as asking for secure code did little, while curated rule sets cut measured risk by up to 70%, by 54% on average for the lower-risk half of models and by 34% for the higher-risk half. Price was a poor guide: a $0.10 run scored lower risk than an $8.17 one, and 15 different models led in at least one of the 11 frameworks tested.

Hill notes that Secure Code Warrior sells AI governance and secure-coding training, that the figures come from its own benchmark, and that rankings can change within weeks of a new release. His practical advice is to write rules for each language, review them, and retest whenever you change models, or in his words, “Treat your AI rules like code.”

AI coding rules are toolchain configuration, and NIST’s SSDF already covers toolchains

An AI coding assistant sits where a compiler sits in the pipeline: it turns a developer’s intent into code that ships. If a text file attached to that assistant changes measured risk by a third to more than a half, the file is a security configuration of a build tool. Software security guidance already has a section for that.

NIST SP 800-218, the Secure Software Development Framework version 1.1, covers compilers, interpreters and build tools under practice PW.6. Task PW.6.2 asks organizations to determine how each tool should be configured, then implement and use the approved configurations. Its examples include testing that the features work as expected, continuously verifying that the approved configurations are being used, and making them available as configuration-as-code. Task PW.6.1 includes following change management processes when deploying or updating these tools, and auditing unexpected changes. Task PS.1.1 asks that all forms of code, configuration-as-code included, be stored under least privilege so only authorized people and tools can change them.

Read with an AI coding tool in place of the compiler, those three tasks give an engineering leader a short list. The rules file lives in the repository with a named owner and restricted write access. A change to it goes through review like any other configuration change. And a change of model, or of model version, counts as a tool update that triggers change management. That third item is the one most teams have no process for today.

A model upgrade behind unchanged AI coding rules is a toolchain update

Teams that pin a compiler version and rerun their test suite before upgrading already know the habit this needs. The equivalent for AI coding is a fixed set of prompts for each language the team uses, written to exercise the weakness categories that matter for its stack. Run the new model with the current rules against that set, scan the output with the static analysis the team already uses, and compare the findings with the last baseline. The rules file and the model version that produced the baseline are recorded together, so a regression points to one of two causes.

This kind of suite is a proven research design. In Asleep at the Keyboard?, accepted at the IEEE Symposium on Security and Privacy 2022, Hammond Pearce and colleagues built 89 scenarios around high-risk weaknesses from MITRE’s Top 25 list and had GitHub Copilot complete them, producing 1,689 programs. About 40% were vulnerable. They also varied the prompts as one of their three test axes, which is the variable a rules file changes. The 2021 model they tested is long retired, and the vulnerability rate says little about current tools. The method still transfers directly: scenario prompts per weakness class, generated code, a scanner, and a count you can compare between runs.

Hill reports Pieter Danhieux of Secure Code Warrior saying the behavior can shift with every new release. A team that retests only when someone remembers to will learn about that shift from production. A team that treats each model change as a release event learns about it from a diff.

AI coding rules fail as a control when they replace review

The same figures set a limit on this argument. An average 34% reduction on the higher-risk half of models leaves roughly two thirds of the measured risk in place, and all of these numbers come from one vendor’s benchmark. Hill also reports, from a forthcoming survey on reviewing AI code, that only 14% of organizations apply clearly defined, consistent standards for reviewing AI-generated code, while 47% rely on the coding agent to review its own work. A rules file written once and then trusted adds a second unverified layer on top of that gap.

The rules file also widens what needs protecting. Because it shapes every line the assistant writes, an edit to it has wider reach than most single commits, which is why PS.1.1’s least-privilege storage matters here as much as the review step. I argued in AI governance is now an engineering problem, published in August 2026, that controls running inside the engineering workflow hold up better than controls kept in policy documents. A rules file is one of the rare governance controls that already lives in the repository. It only works as one if it is measured like one.

Why This Matters for Engineering Leadership

Hill reports a 2026 survey of 591 engineering leaders in which 81% said AI tools or agents already have write access to their repositories, while 32% said their organization has no formal policy for AI agents. Engineering leaders do not need a new framework to close part of that gap. They can treat the rules file and the model version as toolchain configuration under guidance their security teams already follow: an owner, restricted write access, review on change, and a security regression run with a recorded baseline whenever the model changes. That turns the cheapest lever in Hill’s data into a control the team can show evidence for.

FAQ

What are AI coding rules?

They are the instructions a team gives its AI coding tools, usually in a rules file or system prompt, that shape how the model writes code, including which security practices it follows.

How much do AI coding rules reduce security risk?

In Secure Code Warrior’s AI Trust Index, as reported in Hill’s article, curated rule sets cut measured risk by up to 70%, with averages of 54% and 34% on the lower-risk and higher-risk halves of models. The figures come from the vendor’s own benchmark.

Which NIST guidance applies to AI coding rules?

NIST SP 800-218, the Secure Software Development Framework, covers compilers, interpreters and build tools under practice PW.6 and asks for approved configurations, change management on tool updates, and configuration kept as code.

Should a model upgrade trigger a security retest?

Yes. A new model behind the same rules file is a toolchain change, so the team should rerun a fixed set of security test prompts and compare the results with the previous baseline.

Can a rules file replace code review of AI-generated code?

No. Rules change the model’s defaults and lower measured risk. Review and scanning still have to catch what the rules miss.

Source: AI coding rules matter more than your model by Michael Hill, LeadDev, 18 November 2026.

Read more news & analysis