Building a Documentation Skill That Tests Whether Code and Docs Stay Coupled

The most convincing test of our documentation system was a four-line Go file with no documentation. When a pull request added it, the repository caught the documentation drift. That mismatch exposed what the polished reference implementation could not: whether the system would resist drift after setup.

A reference document cannot enforce itself

The starting material was a detailed standard covering repository structure, expected diagrams, and a complete sample implementation. The skill had two jobs. It had to fit that structure around an existing brownfield codebase or bootstrap it in a greenfield repository. It also had to update repository-level instructions such as CLAUDE.md or AGENTS.md and produce the required reference implementation.

Installation alone was not enough. A skill can create immaculate documentation on Monday while the code begins leaving it behind on Tuesday.

I treated code and documentation alignment as repository behavior, not author guidance. The skill installs the structure and instructions. A pull-request workflow then evaluates later changes and reports when governed behavior moves without its associated documentation. This follows the same principle as When Verification Schemas Drift From Runtime Reality: the verification contract must reflect what the system actually does.

Generation failures are visible: a file is absent, a diagram does not render, or the structure is wrong. Governance failures are quiet. The repository continues to build. Reviewers focus on executable behavior. The architecture description becomes historical fiction one reasonable omission at a time.

The deliberately incomplete change

The acceptance case was almost aggressively modest: a pull request added sample/dispatch/internal/probe.go, a four-line Go change, and no documentation. Its stated purpose was to verify that the workflow would comment.

That fixture establishes a narrow but useful fact. The change was intentionally documentation-free, and the pull request exercised the review policy. It proves the shape of the control, not its recall across a large repository.

Most teams test documentation tooling with a positive demonstration. They run the generator, inspect the generated tree, and admire the diagrams. This confirms that the tool produces output when invoked. It says nothing about routine code changes staying aligned with it.

The negative fixture asks the operational question that matters: when an engineer omits the expected documentation action, does the repository notice without depending on that engineer to remember the standard?

A comment is the right first enforcement level. It preserves reviewer judgment and avoids blocking harmless changes while the rules are calibrated. A required check is stronger, but only when its classifications are accurate. If it cannot distinguish an internal refactor from an architectural change, mandatory enforcement turns uncertain policy into dependable irritation.

How a documentation drift check reaches a decision

The workflow needs an explicit path from raw diff to review outcome. I use four stages: normalize the diff, classify changed surfaces, resolve governed documentation, and evaluate the evidence in the same pull request.

First, the check reads file status and hunks rather than treating a list of paths as sufficient evidence. Added, modified, deleted, and renamed files remain distinct. For a rename, it preserves the old and new paths as one change record. It then inspects content changes separately. A pure rename inherits the original surface classification and normally passes. A rename combined with behavioral edits continues through review under the destination path.

Generated files are filtered through repository evidence. That evidence might include generated-file markers, configured output directories, or the generator and source relationship recorded by the installed policy. The check reviews the authoritative source change instead of demanding edits to generated output. If the repository does not identify a file as generated, the workflow does not guess from its appearance.

Next, each remaining code change becomes a surface. It might be a public interface, configuration boundary, runtime component, data flow, operational procedure, or private implementation detail. Path rules provide candidates. The changed symbols and hunks supply the behavioral evidence. A path match alone never proves that governed behavior changed.

The installed policy maps those surfaces to documentation roles, not mechanically paired filenames. For example, a public interface maps to its interface reference. A deployment or runtime-flow change maps to the relevant operational document or diagram. The resolver follows moved documentation through the same rename records, so reorganizing a document does not make it appear absent. If multiple documents plausibly own the surface, the result is ambiguous rather than missing.

Finally, the check looks in the same diff for a substantive update to the resolved document. It can also accept a credible explanation that the documented behavior is unaffected. Any unrelated documentation edit is insufficient. The outcome is a pass when the change is ungoverned, the relevant material was updated, or the explanation addresses the changed surface. Clear omissions receive one evidence-backed comment. Unclear ownership or behavioral impact receives a qualified request for reviewer confirmation.

This calibration accepts tradeoffs. Exempting private refactors avoids the false positive of demanding architectural prose after an internal symbol rename. It also creates a possible false negative. A change presented as private might alter an operational assumption that path rules cannot see. Inspecting hunks and routing uncertainty to a reviewer narrows that gap without pretending the classifier understands the entire system.

The review policy needs a contract

The reusable artifact is the contract below. I use it as the acceptance policy for the skill and pull-request check. It separates mechanical evidence from reviewer judgment.

Repository instructions and pull-request review cover separate failure modes. Updating CLAUDE.md or AGENTS.md guides an agent toward compliant work during a session. Diff inspection catches the cases where those instructions were missed, bypassed, or interpreted differently.

Brownfield and greenfield repositories need different evidence

A greenfield setup can install the prescribed folders, templates, diagrams, and agent instructions directly. Its main risk is premature specificity. Empty diagrams filled with guessed components look complete while encoding decisions nobody has made.

A brownfield setup has the opposite problem. The skill must discover existing documentation, entry points, module boundaries, deployment configuration, and ownership conventions before proposing a structure. Applying the reference tree mechanically can create a second documentation system beside the first. That result is tidy for the skill and worse for everyone else. The discovery problem resembles Discovery Baselining When Nobody Owns the Whole System Map: map the evidence before imposing a new structure.

I rejected a single bootstrap action for both cases. The shared standard defines required documentation roles without assuming identical evidence. Greenfield mode creates placeholders only for known decisions and marks unresolved sections plainly. Brownfield mode maps existing material first. It then proposes moves and gaps for review before writing across the repository.

The extra interaction is part of the cost. Brownfield installation pauses when ownership is ambiguous or sources conflict. That disappoints teams hoping for a one-command cleanup. The friction is cheaper than confidently reorganizing stale documents around an incorrect model of the system.

Where the recommendation breaks down

A review comment adds little when documentation is generated entirely from authoritative source definitions. If an API reference comes from the same schema that drives runtime behavior, another diff heuristic duplicates an existing guarantee. The useful target is human-maintained architectural, operational, or product context that generation cannot recover.

The policy also becomes counterproductive when it treats all code edits as requiring document edits. Alignment concerns represented behavior, not synchronized file churn. Teams need concrete exempt cases and an escape hatch that asks for an explanation instead of ceremonial prose.

The polished implementation proves that the system can generate documentation. The four-line omission proves something harder: after setup, the repository still notices when governed code and its explanation begin to part company.

FAQ

Why is documentation drift hard to catch in code review?

The repository can still build while its architectural or operational explanation becomes stale. Reviewers naturally focus on executable behavior, so a quiet documentation omission can pass without an explicit check.

What code changes should require documentation updates?

Public interfaces, configuration boundaries, runtime components, data flows, and operational procedures are strong candidates. Formatting, pure renames, generated output, and private refactors should usually be exempt unless they alter documented behavior.

Should a missing documentation update block a pull request?

Start with an evidence-backed comment while the policy is being calibrated. Make the check required only when it can classify changes accurately enough to avoid blocking harmless work.

How do you test a documentation drift check?

Add a small, intentional code-only change to a governed surface and verify that the workflow posts one specific comment. Also test pure renames, private refactors, generated files, matching documentation edits, and ambiguous changes.