Run Broker Turns Through a Durable DynamoDB Harness

I was watching a broker process do the one thing a broker process should not do: forget why it had permission to act. The system was supposed to move durable broker execution out of a short-lived ingress path and into an EC2 worker backed by DynamoDB leases. The outcome was good: Slack turns survived restarts, lease expiry, and a controlled kill test, then delivered once. The uncomfortable part was that persistence was the easy bit. The real fix was making scope, checkpoints, event validation, deployment, and delivery identity part of the execution contract.

Why ingress was the wrong owner

Ingress is a fine place to accept a request, authenticate it, normalize it, and put work somewhere durable. It is a poor place to own the lifecycle of a model-assisted turn that may classify intent, gather evidence, call a broker, write events, and notify Slack.

The old shape was too optimistic. A request arrived, Lambda or the edge path did enough work to make the turn feel synchronous, and the rest of the architecture had to pretend that process lifetime and execution lifetime were basically the same thing. They were not. A broker turn has more in common with a payment workflow than a web request. It has claims, state transitions, external side effects, and a requirement that a restart be boring.

The replacement shape was a leased EC2 broker harness. Ingress records the turn. DynamoDB stores queue claims and event history. The broker claims work with a lease, appends canonical events, resumes from the last durable checkpoint, and uses stable delivery IDs when publishing to Slack.

That diagram is deceptively calm. The first implementation had the right nouns and still failed in production twice.

Durable broker execution needs more than persistence

The tempting version of this project is straightforward: put the work item in DynamoDB, claim it with a conditional write, retry on failure, and call it durable. That gets you a durable to-do list. It does not get you durable execution.

The difference is what has to remain true after a process dies.

A resumed broker needs to know which turn it is running, which events have already happened, which external delivery was already attempted, and what authorization scope was effective when the model requested or completed work. Losing that last part is not a cosmetic bug. It changes the meaning of the run.

The first durable harness copied the obvious fields into model and delivery checkpoints. It did not preserve the complete effective scope. The production Slack run completed, but validation rejected model.requested because the scope snapshot was missing. That was the right failure. A validator that accepts a completed run with missing scope is not validation; it is a very polite logger.

The fix was to treat scope as semantic state, not incidental metadata.

The checkpoint had to carry effective scope

The event contract needed to preserve the source kind, resolution status, and visibility on canonical scope events. It also needed to copy the resolved effective scope into model requested, model completed, delivery requested, and delivery completed checkpoints. This follows the same boundary-first logic described in Typed Intent Routing Before an Agent Touches the Broker.

A representative event shape looked like this:

type EffectiveScopeSnapshot = {
  sourceKind: "slack" | "github" | "system";
  resolutionStatus: "resolved" | "partial" | "blocked";
  visibility: "private" | "shared";
  scopeIds: string[];
};
 
type ModelRequestedEvent = {
  type: "model.requested";
  turnId: string;
  checkpointId: string;
  effectiveScope: EffectiveScopeSnapshot;
  createdAt: string;
};

The important part is not the TypeScript. It is the stance. If a later step depends on authorization context, the durable checkpoint has to carry that context in a form the validator can compare.

Snapshot comparison had to be semantic

The next trap was comparing serialized objects as if JSON property order, incidental defaults, and array ordering were the domain model. They are not. The focused harness tests moved toward semantic snapshot comparison: compare the fields that define permission and visibility, normalize where the model permits normalization, and fail when meaning changes.

function normalizeScope(scope: EffectiveScopeSnapshot) {
  return {
    sourceKind: scope.sourceKind,
    resolutionStatus: scope.resolutionStatus,
    visibility: scope.visibility,
    scopeIds: [...scope.scopeIds].sort(),
  };
}
 
expect(normalizeScope(actual.effectiveScope)).toEqual(
  normalizeScope(expected.effectiveScope),
);

This is unglamorous code, which is a compliment. It makes the failure mode legible.

The system was supposed to move durable broker execution out of a short-lived ingress path and into an EC2 worker backed by DynamoDB leases.

The lease is a coordination primitive, not a recovery plan

DynamoDB leases solved one problem: only one broker should own a turn at a time. A conditional write can claim an item if there is no active lease or if the previous lease expired. That is useful. It is not the same as knowing how to resume the work.

A lease can expire while the old process is still limping along. A new broker can pick up the same turn. Slack can already have received a delivery request. The model can already have completed. If the system has no canonical checkpoints, the new worker is left to infer history from vibes and timestamps. Computers are rarely at their best in that mode.

The harness used event storage as the source of truth. Recovery reads the turn history, derives the last completed stage, and continues only from the next legal transition. That separation between observed progress and durable state also matters when you monitor long-running AI agent work without state drift.

The state machine was intentionally small:

The table matters because each row has a recovery decision. Without that, a retry loop is just optimism with a sleep interval.

The deploy contract failed before the architecture did

The most humbling production miss was not a distributed systems edge case. The release did not have zod available at runtime.

The shared core event parser required it. The broker release installed production dependencies incorrectly. The service failed with ERR_MODULE_NOT_FOUND: zod. That is a clean failure, at least. It did not corrupt state. It did remind me that an architecture decision record does not install npm packages, no matter how nicely formatted it is.

The deploy contract changed in two ways: require the lockfile in broker releases, and run a locked production-only install before restarting the service.

{
  "scripts": {
    "check": "npm run lint && npm test",
    "broker:deps": "npm ci --omit=dev",
    "broker:restart": "systemctl restart broker-worker"
  },
  "dependencies": {
    "@aws-sdk/client-dynamodb": "^3.620.0",
    "zod": "^3.23.8"
  }
}

The deployment step became explicit:

cd /srv/broker
npm ci --omit=dev
node -e "import('zod').then(() => console.log('zod ok'))"
sudo systemctl restart broker-worker
sudo systemctl status broker-worker --no-pager

That one-line import('zod') check is not elegant. It is useful. Production dependency errors do not deserve a dramatic detection system when a direct assertion will do.

The validator became the acceptance test

The ADR was not accepted when the design sounded plausible. It was accepted after normal Slack execution and controlled restart evidence. That distinction saved the project from a pleasant lie.

The validator did four things:

1. Started a live broker turn through the normal ingress path.
2. Killed the broker after durable events existed.
3. Waited long enough for lease expiry.
4. Restarted the worker and audited final state, scope snapshots, and Slack delivery count.

A simplified version of the command looked like this:

cd /srv/broker
BROKER_TABLE="agent-events-prod" \
SLACK_CHANNEL_ID="C0123456789" \
node scripts/validate-durable-broker-restart-live.mjs \
  --kill-after=model.requested \
  --lease-expiry-ms=45000 \
  --expect-deliveries=1

The validator was deliberately production-shaped. It did not just unit test the state reducer. It exercised the actual worker, the actual DynamoDB table, the actual service restart, and the actual Slack idempotency behavior with stable delivery IDs.

The final full check passed 256 tests. The focused durable harness suite passed 39. Those numbers are not magic, but they gave the ADR a useful floor: this was not accepted because the happy path worked once while everyone held still.

function deliveryId(input: { turnId: string; checkpointId: string }) {
  return `slack:${input.turnId}:${input.checkpointId}`;
}
 
await eventStore.appendDeliveryRequested({
  turnId,
  checkpointId,
  deliveryId: deliveryId({ turnId, checkpointId }),
  effectiveScope,
});

What I would still watch

This design has costs. DynamoDB event history needs retention policy, compaction, and query discipline. Lease durations need to be long enough for normal work and short enough for recovery to be tolerable. The validator is slower and noisier than a unit test because it touches real infrastructure. That is the price of learning whether restart behavior works outside a diagram.

I would also be careful about treating the broker as a place where all complexity can gather. Moving lifecycle ownership out of ingress was correct here, but a durable worker can become a junk drawer with a service name. The state machine and event contract are what keep it honest.

FAQ

Why is a queue with retries not enough for durable broker execution?

A queue remembers that work exists. Durable broker execution also has to remember which state transitions already happened, what authorization scope applied, and whether an external delivery was already requested.

Where should I capture effective scope in a durable agent workflow?

Capture effective scope on every checkpoint whose later behavior depends on permission or visibility: model requested, model completed, delivery requested, and delivery completed. Store it as semantic state, then validate it after restart.

How do DynamoDB leases help a broker worker?

DynamoDB leases let a worker claim a turn with conditional writes and let another worker recover it after expiry. The lease coordinates ownership, while the event store determines how to resume safely.

How do you prove exactly-once Slack delivery after restart?

Use a stable delivery ID, record delivery.requested before sending, restart from canonical checkpoints, and run a live validator that kills the worker, waits for lease expiry, restarts it, and asserts one completed delivery.

A durable broker is only as real as the restart it can survive without changing the meaning of the work.