Engineering

Privacy-Safe Caddy Access Logs for Production Debugging

August 6, 20266 min read

The dashboard showed zero requests to the route, but our edge records were missing. During the incident, we had no access logging evidence to distinguish “no traffic arrived” from “traffic died before reaching the application.”

TL;DR

Treat access logging as an evidence chain. For Caddy, separately verify that the shipped configuration is accepted and active, a marked request is retrievable, retention actually runs, and collection loss becomes visible. Detect loss with runtime write errors, output freshness, synthetic probe counts, and collector delivery metrics. Until those signals agree, report the interval as unverified instead of calling missing records zero traffic.

Build access logging evidence

The source review found no durable edge record. That observation explained why we could not answer the incident question afterward. It did not prove whether a request had arrived.

Application silence permits several inferences: Caddy may have rejected the request, selected an unexpected handler, or failed before application logging. None is causal proof. The failure mode is a false zero: reporting no traffic when the observation path may be absent or broken.

This applies beyond Caddy. Every observability system must separately prove production activation, end-to-end retrieval, lifecycle enforcement, and visible failure. A parsed configuration or successful query covers only part of that chain.

The operating cost includes persistent storage, restricted operator access, rotation monitoring, privacy review, synthetic probes, and periodic failure injection. The unresolved proof boundary is traffic outside the tested request, time, instance, path, or loss mechanism.

Use this verification matrix as the review contract:

Claim	Test	Proof obtained	Remaining gap
Configuration accepted	Validate with the executable shipped in the image	That executable accepts that configuration	It may not be deployed
Configuration active	Inspect the running instance or exported active configuration	The inspected instance loaded the intended settings	Other instances or later changes remain untested
Event retrieved	Send one marked request and retrieve exactly one matching JSON event	That request crossed emission, storage, and the operator query path	Request volume and continuity remain unproven
Retention enforced	Observe a real roll and deletion of an eligible file	Rotation and deletion worked for that output	Future timing and load may differ
Loss detected	Correlate a sent probe with freshness, write, delivery, and retrieval signals; inject a write failure	The tested loss mode becomes visible	Other loss mechanisms remain untested

Configure privacy-safe records

Keep only fields justified by incident response: time, method, URI path, status, duration, and temporary correlation data. Query strings, cookies, and credentials are unnecessary. Client-address hashes retain correlation value but reduce disclosure rather than anonymize a person.

{$SITE_ADDRESS} {
    log {
        output file {$LOG_PATH} {
            mode 0600
            roll_size 25MiB
            roll_at 00:00
            roll_keep 30
            roll_keep_for 720h
        }
        format filter {
            request>remote_ip hash
            request>client_ip hash
            request>headers>Cookie delete
            request>headers>Authorization delete
            request>headers>Proxy-Authorization delete
            request>headers>X-Observability-Probe delete
            request>uri regexp `\?.*$` ?redacted
            wrap json
        }
    }
 
    reverse_proxy {$UPSTREAM}
}

Caddy’s log directive supports file output, structured encoding, nested-field filters, regular-expression replacement, hashing, and rolling controls. Current releases redact several credential headers by default. Explicit deletion keeps local policy visible if credential logging is enabled elsewhere.

Removing everything after ? discards parameter names and values. That costs diagnostic detail but prevents an unreviewed parameter from entering the record.

remote_ip is the connected peer. client_ip is Caddy’s parsed client address when trusted-proxy handling supplies one. The documented hash is the first four bytes of SHA-256 rendered as eight hexadecimal characters. Collisions are possible, and identical inputs remain linkable while records coexist. Preserving a client IP through trusted proxies explains why those fields are not interchangeable.

Prove configuration and retrieval

Validate with the executable shipped in the deployment image. For a FrankenPHP image:

frankenphp validate --config "$CADDY_CONFIG"

Record the immutable image reference and configuration revision. After deployment, inspect the running instance or exported active configuration. Confirm the filter, resolved persistent output, and rolling options. Changes to an existing file output’s options require a server restart, not merely a configuration reload. The Caddy logging overview also distinguishes HTTP access records from runtime logs.

For retrieval, temporarily copy a random synthetic header into a restricted top-level field:

log_append probe_id {http.request.header.X-Observability-Probe}

Use log_append only for random test data. Any caller reaching the listener can supply the header, so restrict output access and remove the directive promptly.

probe_id="probe-$(openssl rand -hex 16)"
 
curl --fail-with-body --silent --show-error \
  -H "X-Observability-Probe: $probe_id" \
  "$SITE_ADDRESS$PROBE_PATH?synthetic=value"
 
jq -ce --arg marker "$probe_id" \
  'select(.probe_id == $marker)' "$LOG_PATH" > "$PROBE_RESULT"
 
test "$(wc -l < "$PROBE_RESULT")" -eq 1

Inspect the event for time, method, redacted URI, status, duration, hashed addresses, and deleted headers. Then remove log_append, restart or reload as required, inspect active configuration, and confirm a later event lacks probe_id. Expire the probe-bearing record under the approved policy.

Prove retention enforcement

Midnight rotation is clock-aligned, while size-triggered rotation can happen sooner. Thirty rolled files may therefore cover much less than 720 hours under heavy traffic. Both limits govern deletion, and Caddy checks age-based retention when creating a rolled file.

Observe a real roll and later removal of an eligible file. Also verify the resolved path, ownership, numeric mode, mount or backend controls, and retrieval using the investigator’s actual role. Configuration inspection alone does not produce access log evidence of enforcement.

Make loss detection concrete

Monitor four complementary signals: Caddy runtime write errors, output modification freshness under known traffic, synthetic probes sent versus retrieved, and collector delivery or rejection metrics when a collector exists. Freshness alone cannot distinguish a quiet service from a broken output. A scheduled probe supplies known traffic; its sent counter establishes the expectation, and its retrieved counter tests the path investigators use.

Choose the observation window from the probe cadence plus the documented delivery budget. Include the monitor’s evaluation delay when implementing the rule. If no delivery budget has been measured or documented, the alert threshold remains unvalidated; measure end-to-end probe latency before claiming a detection interval.

Minimal vendor-neutral pseudocode:

window = probe_cadence + delivery_budget
 
if sent_probes(window) > retrieved_probes(window):
    alert("access-log probe missing; interval unverified")
 
if known_traffic(window) > 0 and output_age() > window:
    alert("access-log output stale under known traffic")
 
if caddy_write_errors(window) > 0 or collector_delivery_failures(window) > 0:
    alert("access-log delivery failure")

In a safe non-production environment, place $LOG_PATH on a controlled read-only mount. Send the probe, verify that its event does not reach the normal query path, and confirm the appropriate signal alerts within the documented window. Restore writable output and confirm recovery.

That is causal proof only for the injected unwritable-destination failure. It does not prove detection of every delayed, malformed, dropped, or inaccessible event. In the original incident, these signals would not have reconstructed missing history, but they would have prevented an unhealthy logging path from masquerading as a confident zero.

FAQ

Does a matched event prove continuous collection?

No. It proves retrieval for that request at that time. Continuity depends on the ongoing loss signals.

Are hashed client addresses anonymous?

No. They can collide, and repeated inputs remain linkable. Treat them as sensitive data under the applicable policy.

What if rotation has not occurred?

Report retention as configured but not runtime-proven. Observe a safe roll and eligible deletion before claiming enforcement.