I was looking at a verification run that had made it all the way to the scoped EC2 code-change path, which is normally the part where my shoulders drop by about two millimeters. Instead it stopped on codex_code_change_command_invalid. The review layer had produced structured output. The runtime rejected it. That is verification schema drift in its most annoying form: everything looks disciplined until the system that has to execute the claim says, politely, no.
- Pattern
- Executable schema contract
- Output
- Agent operating note
- Use when
- An agent emits structured verification commands that production code later validates
The useful conclusion is narrow: an AI verification contract is not real until the model-facing schema and the runtime validator can fail the same way. If the agent is allowed to emit labels that production code will reject, the schema is no stronger than documentation. I would rather make the contract slightly less expressive and executable than preserve a flexible interface that fails one boundary later.
The incident was small enough to miss
The change looked like a tidy verification PR. It did three things:
- Constrained Codex verification command labels in the output schema to the runtime-safe format.
- Added a contract assertion so schema and runtime validation cannot drift silently.
- Made a destructive chat-integration fixture request an exact Markdown line instead of a fuzzy expectation.
The validation was ordinary:
npm run check
npm run validate:code-changeThe live signal was better than the PR summary. A combined chat-driven run reached the scoped EC2 code-change path, then stopped with codex_code_change_command_invalid. That tells us the failure was not basic routing, missing permission, or a dead integration. The agent got to the right neighborhood, then tripped over a contract mismatch at the execution boundary.
That distinction matters. If routing is broken, you fix routing. If authorization is wrong, you fix the capability envelope. Here the shape was subtler: the agent-facing output schema permitted a value that the runtime validator would not accept. The structured review surface looked healthier than it was.
Where verification schema drift actually lives
A schema is often treated as the source of truth because it is the thing the model sees. In agent systems, that is backwards often enough to be dangerous. The runtime validator is the authority because it decides what will execute.
The schema still matters. It shapes the model's output distribution. It prevents sloppy keys, missing fields, and ambiguous blobs of prose. But if the schema and validator are authored separately, they become two separate policies with similar names.
The failure mode is boring in the way most expensive bugs are boring:
The agent does what the schema allows. The runtime does what production requires. The mismatch is discovered only after the system has spent effort building confidence around an invalid artifact. That is the worst place to find it, because the surrounding evidence makes the invalid command feel less suspicious than it should.
A structured output is not a contract if the executor applies a stricter private law.
The decision: reduce expressiveness at the schema boundary
The tempting fix was to relax the runtime validator. I rejected that.
A reasonable team could argue for it. If historical labels are messy, relaxing runtime validation gets more runs through the pipe. It may also avoid a migration if older artifacts already contain labels outside the safe format. That is a real cost.
But for verification command labels, permissiveness buys little. These labels are not user-facing copy. They are handles used to identify and validate commands. The right property is not literary range. The right property is boring stability across JSON output, logs, fixtures, and runtime checks.
So the schema moved toward the runtime, not the other way around. The agent was constrained to emit only labels the runtime can accept. Then the contract assertion made that relationship executable: if someone later changes the schema without changing the validator, or changes the validator without changing the schema, validation fails before a live run does.
This recommendation stops being correct when labels are part of a public API or a migration surface where old values must remain valid. In that case, the schema and runtime still need an executable contract, but the policy probably becomes versioned compatibility instead of a single tightened format. The point is not austerity for its own sake. The point is that the model should not be invited to produce values production has no intention of accepting.
The fixture needed to stop being suggestive
The chat-integration fixture change was the same class of problem in a smaller costume. A destructive fixture request had been fuzzy. It expected the idea of a Markdown line rather than the exact line.
That fuzziness sounds harmless until the fixture becomes evidence. Destructive actions need more than semantic closeness. They need a testable request shape, especially when the system is deciding whether a command is safe, scoped, and intentional.
Making the fixture exact did not make the system smarter. It made the proof less interpretive. There is a difference. Agent systems already contain enough interpretation in the model call; letting the fixture add another layer is just donating ambiguity to future you, who will accept the gift with resentment.
Verification schema drift review contract
Use this contract when reviewing any agent output schema that feeds a runtime validator.
Scope
This review applies when an agent emits structured verification commands, labels, actions, fixture requests, or execution plans that production code later accepts or rejects.
Required checks
| Check | Pass condition | Fail signal |
|---|---|---|
| Schema mirrors runtime | Every schema constraint has an equivalent runtime rule, or the runtime rule is deliberately looser | Schema accepts a value the runtime rejects |
| Runtime remains authoritative | Production validation decides execution, not model-facing prose | The schema is treated as sufficient proof of executability |
| Contract assertion exists | A test or validation step fails when schema and runtime rules disagree | Rules can drift in separate files without a failing check |
| Labels are runtime-safe | Command labels use the accepted runtime format and avoid display-only text | Labels contain spaces, punctuation, or casing the runtime rejects |
| Fixtures are exact | Safety-sensitive fixture expectations use exact Markdown or exact structured values | A fixture accepts a paraphrase for a destructive request |
| Live failure names boundary | Errors identify whether rejection came from schema, routing, permission, or runtime validation | One generic failure hides the rejected layer |
Review question
If the agent emits the most permissive value allowed by the schema, will the runtime accept it without a special case?
If the answer is unknown, block the change until the schema and runtime are tested against the same examples.
Minimum validation
Run the normal type and lint checks, then run the code-change validation path that exercises the agent output schema and runtime validator together.
npm run check
npm run validate:code-changeMerge condition
Merge only when a future disagreement between schema and runtime rules would fail validation before a live agent run reaches the execution boundary.
The cost of making the contract executable
The sharp edge is maintenance friction. Once the assertion exists, small schema changes are no longer local. A developer who wants to rename, widen, or prettify a label format has to understand the runtime rule too.
That is mildly inconvenient. It is also the point.
The contract moves a class of failures from live runs into review. It does not remove judgment. It does not prove that the command is useful, safe, or sufficient. It only proves that one specific boundary is honest: the thing the agent is allowed to say is a thing the runtime is prepared to hear.
For agent verification systems, that is a modest claim. It is also the kind of modest claim worth enforcing, because the alternative is a review layer that looks structured right up until production refuses to participate.