The request sounded harmless in the way expensive product ideas often do: could we use commits to generate stories for marketing? We already had agents around the software delivery workflow, and commit story mining sounded adjacent enough to be tempting. The trap was that the obvious architecture, bulk GitHub indexing, would have optimized for the easiest thing to store rather than the hardest thing to recognize.
- Pattern
- Real-time narrative signal over bulk code volume
- Output
- Agent operating note
- Use when
- A team wants AI-generated stories from engineering activity and is considering repository indexing as the first architectural move
The useful decision was to park repository-wide indexing for this use case. Not because indexing is bad. It is very good when the question is code health, dependency exposure, vulnerability search, refactoring candidates, or cross-repo lookup. But for editorial story discovery, indexing every changed line treats a meaningful commit and a 100k-line deletion as equivalent raw material. They are not equivalent. One may contain an engineering decision. The other may be mostly compost.
The mistake hiding inside the reasonable ask
The concrete incident was small, which is usually how these decisions sneak in.
A marketing stakeholder wanted commit-derived content. The idea was sensible: engineering work produces real stories, and commits are closer to the work than a quarterly recap or a polished release note. We also already had an internal agent surface doing useful delivery-adjacent tasks: code reviews, daily status note collection, and triage. So the shortest mental path was obvious. Index GitHub, let the model search across history, and ask it for story candidates.
That path was also wrong for the job.
The direct observation was that no GitHub indexing was active yet, and the current agent work was narrower and more operational. The inference was that adding a generic indexing layer would create an impressive amount of searchable material without necessarily improving story selection. The decisive constraint was diff shape. Large diffs, including examples on the order of 100k-line deletions, would swamp the system with low-value context if we treated repository changes as a flat corpus.
That matters because editorial usefulness is not proportional to line count. A ten-line change can reveal a customer constraint, a design reversal, or a reliability lesson. A massive deletion can be a dependency cleanup, generated artifact removal, or mechanical churn. Sometimes that is a story. Often it is just a receipt.
The engineering consequence was architectural, not cosmetic: if we built bulk indexing first, we would have made the retrieval problem look solved while pushing the harder judgment problem downstream into prompts and manual review.
Why commit story mining is not code search
Code search wants recall. Story mining wants selectivity.
Those two pressures produce different systems. A code search system is rewarded for remembering as much as possible and retrieving the right shard later. A story system is punished when it turns every change into a candidate, because the human review surface becomes a landfill with filters. The failure mode is not that the model cannot summarize a diff. The failure mode is that it summarizes the wrong diffs with confidence and asks an editor to care.
For story generation, the object worth capturing is not the line change alone. It is the relationship between intent, review discussion, commit timing, and the operational shape of the change. A real-time agent can observe a commit as part of a live workflow and ask better questions while the context is still warm:
- Was this change tied to a customer pain, incident, launch, migration, or constraint?
- Did the review reveal a tradeoff that is understandable outside the team?
- Is the interesting part the code, the deletion, the rollback, the simplification, or the decision not to proceed?
- Would a staff engineer recognize this as a transferable lesson rather than a changelog item?
Bulk indexing can preserve artifacts. It cannot, by itself, preserve why anyone should read about them.
The staff-level tradeoff
Parking indexing gives up some real things.
You lose broad historical recall. You cannot later ask the system to mine every repository from the last two years unless you build that path separately. You also lose the comforting feeling of having one substrate for many future use cases. That comfort is not nothing. Shared infrastructure is often the difference between a useful prototype and another little agent wandering around with a clipboard.
But the alternative we rejected, bulk indexing as the first move, front-loads storage and retrieval before proving that the candidate selection loop works. That is backwards for editorial story generation. The bottleneck is not access to more lines of code. It is deciding which engineering moments deserve human attention.
The recommendation stops being correct under a few conditions. If the primary use case becomes security, dependency intelligence, architecture search, or refactoring analysis, indexing moves back toward the center. If the team needs retrospective mining across a large historical corpus, a batch index may be necessary. If commit metadata is too sparse and the live workflow has no reliable review or status signals, a real-time observer may not have enough context to judge intent.
For the stated use case, though, the better first system is a narrow observer: watch commits as they happen, attach nearby context, score for narrative signal, and hand only a small number of candidates to a human.
The first product question is not "Can we index the repository?" It is "What would make this commit worth interrupting an editor?"
A review contract for story candidates
The reusable artifact is the gate I would put in front of any agent proposing engineering stories from commits. It is deliberately stricter than a summarizer prompt. The point is to make the agent prove the candidate has narrative signal before it creates editorial work.
You are evaluating whether an engineering change is worth sending to an editor as a story candidate.
Use only the supplied commit metadata, diff summary, review notes, status notes, and linked work item context. Do not invent customer impact, metrics, incidents, outages, or motivations.
Return one of three decisions: send, hold, or discard.
A candidate may be send only when all required fields below are supported by evidence.
| Field | Requirement |
|---|---|
| Change shape | State whether the change is feature work, cleanup, deletion, migration, bug fix, reliability work, or other. |
| Human intent | Identify the stated reason for the change from review notes, status notes, or linked work item context. |
| Narrative hook | Name the engineering tension a reader could learn from. This must be more specific than "we changed code." |
| Evidence | Cite the supplied artifact type that supports the hook, such as commit message, review note, status note, or work item. |
| Exclusions | Name any missing evidence that would make the story risky or speculative. |
| Editorial cost | Estimate reviewer effort as low, medium, or high based on missing context and sensitivity. |
Decision rules:
- Use
discardfor mechanical churn, generated file changes, dependency-only updates, or large deletions with no stated intent. - Use
holdwhen the change looks potentially interesting but the reason, consequence, or audience value is unsupported. - Use
sendonly when the engineering tension is clear and the evidence is strong enough for a human editor to evaluate without reconstructing the whole project.
Output format:
Decision: send | hold | discard
Change shape:
Human intent:
Narrative hook:
Evidence:
Exclusions:
Editorial cost:
One-sentence pitch:
This contract does two things that a bulk index does not. It forces the agent to classify the shape of the change before it tries to sell the story, and it creates a respectable answer for "no." That second part matters. Without a first-class discard path, every agent becomes a generator of plausible maybes, and plausible maybes are where editorial time goes to quietly expire.
What I would build first
I would start with a real-time commit observer attached to the delivery events the team already trusts. Not a grand archive. Not a universal memory layer. A small loop:
- Observe a commit, pull request, or merge event.
- Collect nearby intent signals from review notes, work items, and status updates.
- Run the candidate contract.
- Store only the decision, supporting evidence references, and a compact summary.
- Let a human accept, reject, or annotate the candidate.
The important part is that the first durable record is not the full diff. It is the agent's claim about why the change might matter, plus the evidence needed to challenge that claim. Raw code can remain reachable through GitHub. The product database should preserve editorial judgment.
That design also creates better future indexing options. If broader GitHub indexing becomes necessary later, the team can index with labels learned from the real-time loop: which kinds of commits became stories, which were discarded, which needed human context, and which large diffs were actually meaningful. That is a better training signal than treating every token in a repository as equally promising.
The sharp edge
The cost of the real-time approach is that missed moments may stay missed. If the observer is offline, the webhook is misconfigured, or the review context lives somewhere the agent cannot access, the system may fail silently unless monitoring is part of the design. Bulk indexing is more forgiving on capture because history can be reprocessed.
There is also a human behavior cost. Engineers and reviewers need to leave enough intent behind for the observer to work. If every commit says fix stuff and every review says lgtm, the agent has very little to mine. The architecture can reduce editorial noise, but it cannot manufacture a culture of legible engineering decisions.
That is still the better bargain for this use case. A story-mining system should be judged by the quality of interruptions it creates, not the volume of code it can remember. When the goal is editorial signal, the first architecture should watch for meaning at the moment it appears.