Measuring Performance Work Before Declaring Victory
Thirteen performance PRs shipped together, and production became three to six times faster. Then performance attribution failed the question that mattered: which change had actually worked?
TL;DR
Frequently used pages became roughly three to six times faster, with transaction p95 ranges moving from about 0.7–3.6 seconds to roughly 80–500 ms. That justified retaining the release long enough to investigate, but not repeating any individual optimization elsewhere. To decide whether to retain, revert, or repeat one change, compare its old and new paths under a controlled rollout, match route latency to database-counter deltas, and account for confounders and shifted costs.
We had removed correlated counts, narrowed ORM includes, moved aggregation into SQL, adjusted cache behavior, added indexes, and resized memory. Each mechanism was plausible. The failure mode was bundled-change attribution: the release window supported a bundle-level observation but no allocation of credit among its changes.
The operating cost extends beyond an imprecise retrospective. Weak attribution can preserve ineffective code, spread an optimization whose mechanism was never demonstrated, conceal index write and maintenance costs, or reduce trace sampling before rare regressions are understood. Attribution quality determines what we retain, what we revert, and what is worth repeating.
Performance attribution needs a controlled comparison
A useful follow-up experiment sends comparable requests through old and new implementations during one bounded period. A feature flag, canary, or deterministic cohort can select the path, provided both cohorts encounter comparable operation mix, data distribution, cache policy, worker capacity, database resources, and background load. Record the release identifier, observation timestamps, routing rule, request counts, errors, and tracing rule.
For the SQL aggregation change, keep both paths available and label route telemetry by path. Capture p50 and p95 route latency for each population. Read p50 beside p95: the median shows ordinary behavior, while p95 can expose slow-path movement but becomes unstable at low volume. Counts and error rates must accompany both because different traffic or disappearing timeouts can make completed requests look faster.
Sentry sampling is part of the measurement contract. We temporarily raised transaction sampling to 100% while investigating missing transactions and ranking slow operations. That improved visibility while increasing ingestion, network traffic, SDK processing, storage consumption, and billing exposure. Later sampling reductions lowered those costs but weakened evidence for rare operations and low-traffic accounts. Record the rule with every comparison; Sentry documents the controls in its JavaScript tracing guide.
The decision path is causal, not merely chronological:
Calculate exact-query snapshot deltas
pg_stat_statements exposes normalized query identity, calls, rows, and execution-time aggregates. Its columns and behavior are documented in the PostgreSQL reference. Take snapshot A immediately before the bounded comparison and snapshot B immediately after it. For each path, select the exact normalized statement identity, including queryid, dbid, and userid where those dimensions distinguish entries.
Calculate:
calls_delta = B.calls - A.calls
total_exec_time_delta = B.total_exec_time - A.total_exec_time
rows_delta = B.rows - A.rows
mean_exec_time_delta = total_exec_time_delta / calls_deltaThe final value is the interval mean, not B.mean_exec_time - A.mean_exec_time. Treat zero or negative calls_delta as an invalid comparison. Also reject the delta if counters reset, the statement was evicted, its identity changed, or relevant server state invalidated the boundary. A coordinated pg_stat_statements_reset() can create a boundary, but it removes shared diagnostic history; timestamped snapshots are usually safer.
Parameter normalization can conceal a large account or unusual permission scope behind an acceptable mean. After ranking statements by interval total time, inspect representative and worst-case plans with appropriate parameters. Do not run invasive EXPLAIN ANALYZE operations against expensive production writes without understanding their effects.
In the incident, moving aggregation into SQL reduced rows crossing the application boundary by roughly three to five times. That is a mechanism observation, not a measured share of the aggregate latency improvement.
Decide whether to retain, revert, or repeat
The source provides no predeclared numeric threshold for a meaningful win, so inventing one after seeing the result would be false precision. Before the controlled rollout, choose a latency threshold based on baseline variability and the operational requirement, plus an acceptable error bound and minimum evidence volume. Then apply this review contract:
| Evidence | Retain or repeat | Revert or revise | Repeat the test |
|---|---|---|---|
| User effect | New-path p50 or p95 clears the predeclared threshold with comparable counts and errors | Latency or errors materially worsen | Volume is insufficient or populations differ |
| Mechanism | Exact-query interval calls, rows, and execution time move in the expected direction | Counters contradict the proposed mechanism | Identity, reset, or eviction breaks the delta |
| Confounders | Traffic, data, cache, capacity, jobs, and sampling are comparable | A known condition explains the apparent gain | A material imbalance cannot be bounded |
| Shifted cost | CPU, write load, storage, vacuum, and maintenance remain acceptable | The gain merely moves unacceptable cost elsewhere | Cost observation is missing or too short |
Indexes for permission filters over two large relations illustrate shifted cost. They may lower reads while increasing storage, cache pressure, write amplification, vacuum work, and maintenance. CREATE INDEX CONCURRENTLY avoids blocking writes in common production cases, but cannot run inside a transaction block, performs extra work, and can leave an invalid index after failure. Deployment automation must check validity and handle failed artifacts deliberately.
Derived counts can likewise shorten reads by moving work into writes, corrections, imports, retries, reconciliation, and backfills. Until drift detection operates, source rows remain authoritative and consistency risk stays in the decision.
Our observation was that the bundled release coincided with aggregate latency improvement and that SQL aggregation transferred fewer rows. The inference is that several shipped mechanisms were plausible contributors. Causal proof for any one PR remains unresolved because the original window did not isolate it. The controlled comparison above is the next test; no portion of the aggregate improvement should be assigned to one PR.
The next chapter applies the same discipline to Django cache performance, where hit and miss observations still require matched workloads.
FAQ
What if p95 is sparse?
Report counts and raw slow observations. Extend the window only when it does not cross a material deployment or traffic regime; otherwise repeat the bounded test.
Should database counters be reset?
Usually use timestamped snapshots. Reset only with coordination and a recorded boundary because it removes shared diagnostic history.
What if derived data drifts?
Treat source rows as authoritative, reconcile the derived value, and inspect idempotency and missed write paths. Include consistency risk until drift detection is operating.