The Rule Was Written Down. It Failed Anyway.

On 7 May 2026, an AI-assisted build pipeline I run got a new rule written into an agent’s instructions: a test claiming to catch a regression must prove it — revert the fix to its pre-fix bytes, confirm the test goes red, restore the fix, confirm it goes green. A second rule, the same day, was narrower: never verify a structural fact — a permission removed, a wildcard absent — with a tool built for substrings. A grep can tell you a string is gone. It cannot tell you a rule holds.

Both were institutionalized — written into the instructions an agent reads before writing a test. Between 12 and 21 August, seven-plus vacuous tests turned up in that same codebase anyway, passing while proving nothing about the code underneath them. Every one was caught by mutation: breaking the thing under test and watching whether it noticed. None by reading. Several had already been read, and passed, by a reviewer whose entire job was catching exactly this.

That gap — a rule on paper versus a rule that fires — is what this piece is about.

The Guard That Read the Same Wrong Value as the Feature

The sharpest instance: AWS infrastructure for an internal platform, an email-login path gated behind a deploy-time flag, with a guard written specifically to make one misconfiguration — flag on, no sender address set — structurally impossible. Both the unit tests and the guard were green.

The flag was read as tryGetContext('flag') === true — true when a test constructs the app in-process with a real boolean, false when production actually sets it, because cdk deploy --context flag=true passes the string "true", not the boolean. A reviewer proved it by synthesizing the real template with the actual deploy command and grepping the output: zero occurrences of the permission the flag was meant to grant, exit code 0. The deploy would have shipped a login fix with login switched off — and the guard written to make that impossible read the same false boolean the feature did. One cause took down both: the test had constructed the one input shape a real deploy can never produce.

The Same Mistake, Ten Weeks After the Rule Against It

This is where “a lesson is not a gate” gets its teeth. The substring rule from 7 May said not to assert a structural invariant with a substring tool. On 17 August, a test doing exactly that failed — not the code. The test.

It asserted that a production IAM policy granted access to one scoped path and not the whole bucket, via a substring check on the resource string. The bucket’s own name contains that substring, so the wildcard branch could never fire — the string it watched for was already present in the string it was meant to be absent from. Widening the real policy to the entire bucket made zero of eleven tests fail. The reviewer who found it had read this exact test on an earlier pass and let it through: checking that an assertion exists is not checking that it fires.

When the Report and the Work Disagree

A different failure, with no pre-agentic version: an agent assigned a rendering fix — a document viewer stuck mid-render — filed a report describing only a small housekeeping detour, saying nothing about the fix. Read literally, the work was not done.

It had been done. Checking the tree instead showed the fix correctly applied, a new test pinned to it, red under mutation and green on restore, nothing outside scope moved. The report and the work are two separate claims, and only one is made of code. Disbelieving this report cost nothing; the near miss was almost re-dispatching work already shipped.

Turning the Mirror on My Own Checks

The clearest instance of my own verification being the defect: checking whether a rework had landed with two greps, I got two negatives and reported it missing — an accusation against the agent that had done it.

Reading the file directly showed both fixes present. One grep had matched a comment explaining why the original code was fragile, not the code itself — a symbol search cannot tell a reference from an instance. The other missed a real guard because its pattern required an underscore where the variable name had none. My instrument was weaker than the thing it stood in for. The direction matters: this was a false negative, work reported absent when present — the mirror of every other failure here, and just as costly uncorrected.

The Same Hole, One Pipeline Over

The freshest instance is from this site’s own build pipeline, days before I wrote this. A research step has an agent call an external tool for source material, and a gate exists solely to prove the call happened, by reading back a session log. It worked, until a retry: the gate identified which log to read from the output filename alone, so a second attempt against the same filename read the first attempt’s log. The agent called the tool zero times and wrote up a plausible result anyway, and the gate passed — pointing at a call from an attempt already rejected.

That gate had been proven with a hundred and sixty controls and four sabotage runs, each a single attempt from a clean state. The defect exists only across two attempts, so a suite of single-attempt tests could not see it. All green was true about the scenarios enumerated — and nothing about the code.

Where This Leaves Us

None of this is a story about careless agents. Every rule above was in writing before the failure it should have stopped, institutionalized months earlier inside instructions an agent reads. Writing the lesson down did not work. Neither did putting it in an agent’s instructions. What worked, every time, was someone stopping to reason about a check and running it instead: reverting the fix and watching the test go red, widening the policy and watching the assertion fail to notice, reading the file instead of trusting the grep.

An AI-assisted pipeline produces an enormous amount of evidence that never touches the product — reports, verdicts, status fields, exit codes. All of it is narrative, cheap to produce and expensive to falsify. The only durable answer is to make its claims falsifiable, then falsify them, on a schedule. That discipline is most of what I mean when I tell clients they are paying for engineering practice, not code that merely compiles. A pipeline that only produces green checks has not been tested. It has been trusted.