The config diff looked harmless until the live broker produced four fallback cases. We had centralized the default model/profile mapping, reviewed the profile names, and exercised the path, but the runtime was still quietly taking routes the PR claimed to control. That gap is the thing I want to make concrete, because I have watched teams relax at exactly that moment and let the runtime absorb the edit without bound evidence.

I do not think that is good enough for shared AI platforms. If a default model change can alter prompts, profiles, approvals, and fallback behavior, it needs versioned sessions as a regression contract before the default moves.

Do not approve AI runtime drift by reading the diff alone. Bind the changed defaults to versioned sessions, deterministic reports, live evidence, and CI failure rules. Otherwise the system can look tidier while quietly becoming less predictable.

The incident was not a bad model

The concrete case was a shared agent system used to route work through reviewed execution profiles. The work was mundane and more dangerous than it looked: centralize reviewed EC2 profile and model defaults in production config.

That sounds like cleanup. It was not. A default is a policy decision pretending to be a constant.

The production config selected which model/profile combination the broker should use for a frontend request. The profile contract also carried approval requirements, which meant the change could affect both what the agent did and when it needed human sign-off. During live evaluation, four fallback cases appeared. One specific instance: a frontend-model case that expected a named execution profile received a generic fallback route instead, because the centralized default did not yet cover that entry point. The deterministic report accepted the config shape, but the live broker took a path the PR had not accounted for.

The fix was not another reviewer note in the PR. Making the PR unable to land unless the evidence moved with the runtime surface was the actual fix.

The final deployment evidence recorded a merged release, a shared CI run, a final command in the runtime environment, and a post-deploy team notification. The live report closed at 9 of 9 frontend-model cases with no fallback. That last phrase matters because it names the behavior the system was allowed to claim, not just the command someone ran.

A versioned session is a contract, not a sample

Session-based regression gets weak when teams treat the sessions as example transcripts. A transcript is inert. It can be sanitized and still irrelevant to the runtime behavior being changed.

In this case, the sessions had a job: bind together four things that otherwise drift independently.

Once a default influences the runtime behavior of multiple frontend-model cases, it cannot be treated as plain config. The corpus, profile policy, and live report have to move together, or CI fails.

A team might object that this slows down model upgrades. It does. That is the trade. You give up the ability to patch a shared default as casually as changing a feature flag. In exchange, you get a record that a specific corpus, contract, and runtime path agreed at the moment you changed the default.

This contract is not appropriate for every change. If the model choice is purely local, reversible by one developer, and not part of a shared approval or production execution path, the overhead is not justified. For a scratch tool, the gate is theatrical. For a broker that mediates production-capable agent work, letting a config edit carry policy without evidence is the theater.

What the checks actually verified

The review rule we adopted: a default model change must prove that deterministic policy and live execution still agree.

The deterministic layer verified the contract: which profiles exist, which approvals they require, which model defaults are selected, and whether the session corpus expects those cases to run under those constraints. It did not try to predict every token the model would produce. That would be brittle and comforting in the wrong way.

The live report answered what deterministic checks could not: when the broker receives the real frontend-model cases, does it execute the intended route without fallback?

Fallback became a first-class regression signal because of this structure. Without it, the team could have centralized defaults, passed ordinary tests, and shipped a broker that silently substituted behavior in the very cases the change was meant to control. The system would still be "working" in the least helpful sense of the word.

The review contract

The artifact below is the reusable part. It is not a prompt about writing better PR summaries. It is a gate for deciding whether a shared AI runtime default can move.

This is heavier than a normal config review, and it is supposed to be. The cost is maintaining the corpus and refreshing reports when the contract changes. CI can also become noisy if the sessions are too broad or if reports encode incidental details. The discipline is to make the corpus cover policy-relevant behavior, not every conversational flourish the agent might produce.

What we rejected

The tempting alternative was to rely on the centralized config plus ad hoc live runs. That would have been faster and probably acceptable for a single-operator tool. It failed here because the change crossed several ownership surfaces: runtime defaults, approval requirements, CI, and deployment evidence.

Keeping deterministic checks only was the other option. That would have caught schema and contract drift, but it would not have caught the four live fallback cases. The live broker is where routing, environment, packaging, and configuration meet. A deterministic report cannot fully represent that path, and trusting it to do so is how teams end up with a green check attached to a different system than the one users hit.

The final version used both. CI failed when the corpus, profile contract, or model defaults drifted without matching evidence. The deployment record then closed the loop with merged-release provenance and a live validation across all nine frontend-model cases.

The uncomfortable part

This practice makes the PR less forgiving. A small config edit now carries paperwork, and some of that paperwork has to be refreshed by people who understand the runtime. It will feel bureaucratic if the team has not agreed which defaults are policy-bearing.

The way out is to classify defaults honestly, not to weaken the gate. Local defaults can stay light. Shared production defaults need a regression contract. Approval-bearing defaults need live evidence. Treating them as the same category is cheaper only until the first quiet fallback.

The goal is not to make model changes rare. It is to make them legible: a reviewer should be able to say what changed, what evidence supported it, and what behavior the runtime is now allowed to claim.

FAQ

Q: Does every model change need this full process?

No. The contract is scoped to shared defaults that carry policy weight: production broker routes, approval-bearing profiles, or any default that affects multiple entry points. A change that is local, easily reversed, and owned by one developer does not need a versioned corpus or live report.

Q: What counts as a fallback case?

A fallback case is any frontend-model pairing where the broker did not execute the intended named profile and instead took a generic or secondary route. In the incident above, one case fell back because the centralized default did not cover that entry point. Nonzero fallback in the live report is a merge blocker unless the PR names the case, explains the cause, and records an owner decision.

Q: What goes in the versioned session corpus?

The corpus should cover the policy-relevant combinations: each frontend entry point paired with the expected execution profile and model default. It does not need to capture every possible conversational output. Sanitized real cases from staging or production work well as a starting set; synthetic cases are acceptable if they cover the boundary conditions the contract actually cares about.

Q: What if the deterministic report passes but the live report has fallback?

That is the scenario this gate exists to catch. The deterministic report verifies contract shape; it cannot verify runtime routing under real environment conditions. When the two disagree, hold the PR, diagnose the fallback path, and update the corpus or config before re-running both reports.

Q: How do we keep the corpus from going stale?

Tie corpus refresh to contract changes, not to calendar. When a profile is added, renamed, or its approval policy changes, the corpus update is part of the same PR. CI should fail if the corpus version predates the config change being reviewed.