Turn Painful Agent Runs Into Evaluated Skills

A painful agent run is not automatically reusable knowledge. Most transcripts preserve the route that happened to work, mixed with false starts, environment accidents, and assumptions the operator never wrote down. Turning that into a "skill" without evaluation only makes the accident easier to repeat.

The standard I use is stricter: the second run should be cheaper and more reliable than the first, on an input the first run did not see.

The number that changed with the boundary

In one read-only investigation, the agent returned a precise-looking count of redundant directories. The number changed when the search boundary changed. The first result was not fabricated, but it bundled an unstated definition of "redundant," a particular traversal depth, and exclusions that existed only in the conversation.

That run contained a useful procedure. It did not yet contain a reusable skill.

The distinction matters because a polished runbook could have fossilized the original counting mistake. The real reusable knowledge was how to define the candidate set, preserve exclusions, compare content and ownership, and report uncertainty before recommending deletion.

A skill has four maturity levels

I separate reuse into levels so a team does not call every saved prompt an engineering capability.

Most teams stop at Level 1 and then discover, during an urgent second run, that the procedure depended on hidden machine state or the original author's judgment.

Extract invariants, not chronology

The transcript tells you what happened. Skill design asks a different set of questions:

• Trigger: What observable condition should invoke this skill?
• Inputs: Which values vary per run, and which context must be fetched?
• Invariants: Which checks must happen regardless of input?
• Authority: What may the skill read, mutate, deploy, or delete?
• Evidence: What output lets a reviewer verify the result?
• Failure exits: When must it stop rather than improvise?
• Freshness: Which assumptions can expire between runs?

For the directory investigation, a reusable contract might look like this:

name: redundant-directory-audit
mode: read-only
inputs:
  root: required
  max_depth: required
  exclusions: required
checks:
  - list candidates within the declared boundary
  - compare file content or hashes
  - inspect references and ownership
  - check recent activity
output:
  - candidate path
  - reason for suspicion
  - conflicting evidence
  - confidence
  - deletion recommendation
stop_when:
  - boundary is ambiguous
  - ownership cannot be established
  - comparison requires write access

The output does not promise deletion. It produces a review set. That is the invariant the original run was trying to discover.

Evaluate the dangerous edges

A skill should be tested against cases designed to break its assumptions, not just a replay of the successful transcript.

For a repository cleanup skill, I would keep fixtures for:

1. Two byte-identical directories where one is actively referenced.
2. Similar names with materially different content.
3. A generated directory whose source lives elsewhere.
4. An apparently unused directory with recent commits.
5. An inaccessible path that must be reported as unknown, not empty.

The evaluation should score the decision artifact, not merely whether the tool ran. Did it preserve uncertainty? Did it avoid recommending deletion when ownership was unresolved? Did the same input produce a stable candidate set?

For model-backed skills, keep the raw evidence and expected decision separate from the phrasing of the answer. Models and prompts will change. The operating constraint should survive them.

Reuse has maintenance cost

Packaging a workflow creates an owner and a compatibility obligation. APIs change, repository layouts move, permissions tighten, and a once-valid heuristic becomes dangerous. A skill that is not versioned and exercised becomes a confident form of stale documentation.

Not every painful run deserves this investment. I extract a Level 2 or Level 3 skill when at least one of these is true:

• the task will recur across teams or environments;
• failure has production, security, or deletion risk;
• the discovery cost was high enough that repeating it is wasteful;
• the procedure enforces a team policy that should not depend on operator memory.

Otherwise, a short decision record may be the better artifact.

The durable output of an agent session is not always code. But it is only a skill when it can face a new case, preserve its limits, and still produce a result another engineer can trust.