Monitor Long-Running AI Agent Work Without State Drift

A long-running agent monitor should be boring. It should read the same evidence, compare it with the previous checkpoint, report the delta, and stop under known conditions. The moment it starts fixing what it observes, changing its query, or retelling the full history, it is no longer a monitor.

The core design constraint is observational integrity: repeated checks must not silently alter the experiment they are meant to measure.

The useful output was one line

After a production rule change in the source sessions, an agent checked metrics every fifteen minutes for a fixed bake period. Each cycle ran the same read-only query, compared key rates with the prior checkpoint, and flagged named anomaly classes. When nothing changed, it returned a compact no-change status instead of replaying the investigation.

That restraint is the feature. Long-running agent work tends to drift because each cycle receives more narrative state, interprets the task again, and is tempted to take an action that makes the next result harder to explain.

Define a stable evidence surface

Before starting a heartbeat, specify the observation as if it were a small API:

cadence: 15m
duration: 2h
mode: read-only
evidence:
  query: traffic_rule_outcomes
  window: rolling_15m
  dimensions:
    - total_requests
    - matched_requests
    - false_positive_sample
checkpoint:
  store:
    - observed_at
    - query_version
    - values
report:
  unchanged: "No material change. No action applied."
  changed: "Delta, threshold status, and source timestamp only."
stop:
  - bake period complete
  - anomaly threshold crossed
  - evidence query fails twice

The query_version matters. If the agent changes filters or time windows halfway through, the values are no longer comparable. A monitor may discover that its evidence is inadequate, but it should stop and propose a revised contract rather than silently changing the measurement.

Separate checkpoint state from narrative memory

The minimum state between cycles is structured:

{
  "observedAt": "2026-07-11T12:15:00Z",
  "queryVersion": "traffic-rule-v2",
  "values": {
    "totalRequests": 18420,
    "matchedRequests": 37,
    "confirmedFalsePositives": 0
  },
  "thresholds": {
    "falsePositives": 1,
    "matchRateChangePct": 20
  }
}

The next cycle does not need the entire conversation. It needs the contract, previous checkpoint, current result, and stop conditions. This reduces state drift and makes a restarted monitor equivalent to the original one.

For high-value monitoring, store checkpoints outside the model context. An append-only record makes missed cycles, duplicated runs, and changed queries visible.

Idempotence is not optional

A read-only command can still have side effects. Authentication may refresh credentials, a dashboard export may create records, and a status endpoint may trigger lazy work. Verify the behavior of the evidence source before declaring the monitor read-only.

Each cycle should also be safe to repeat. Schedulers duplicate jobs, network retries happen, and agents restart. Use a checkpoint key such as (monitor_id, scheduled_at, query_version) so a repeated cycle updates or compares the same observation rather than inventing a second timeline.

When an anomaly appears, the monitor should preserve the result and stop or escalate. Remediation belongs to a separate authority envelope. Otherwise, the monitor may change the system before a reviewer sees the evidence that triggered it.

Alert on decisions, not movement

Reporting every numeric change defeats the purpose of an agent layer. The contract should distinguish:

• normal movement that needs no message;
• a material delta worth recording;
• a threshold breach that requires review;
• loss of observability, which is itself an anomaly.

Thresholds should map to decisions. "Error rate increased" is incomplete. "Error rate exceeded the rollback threshold for two consecutive windows" tells the operator what the policy expects.

There is a cost to this discipline. Fixed cadence can delay detection, rolling windows can hide short spikes, and compact delta reports can omit context needed to interpret a novel failure. Keep direct dashboards and raw queries available for drill-down. The heartbeat is a control protocol, not the entire observability system.

The handoff at the end

A monitor finishes with one of three outputs:

1. Completed: the terminal condition was reached, with baseline-to-final delta and no action applied.
2. Escalated: a named threshold or observability failure occurred, with the triggering checkpoint attached.
3. Expired: the monitoring window ended without enough evidence, with a recommendation for the next contract.

"Still watching" is not a terminal state. Every long-running agent task needs a clock, a stable observation surface, and a point where responsibility returns to a person or another explicitly authorized stage.