Replacing a Disappearing Sentry With an End-to-End Datadog Observability Plane

Next week, the team loses access to Sentry. When the first customer-visible failure hits, can the on-call operator complete a tested investigation from Cloudflare through Datadog and page the right owner, or will the trail break between systems?

Replace parity with a tested investigation

The existing Sentry installation cannot be inspected. Its application coverage, alert rules, routing, retention, historical rates, and issue groupings are unknown. Once access disappears, anything stored only there becomes unavailable.

That makes parity unknowable. Feature checklists compare configuration; a tested investigation proves whether an operator can start with a symptom and terminate in action by an accountable owner. This narrower migration claim remains useful whenever a retiring system cannot provide a trustworthy baseline.

Reconstruct requirements from evidence that remains: application configuration, deployment definitions, runtime inventories, cloud resources, and conversations with accountable owners. For each user-visible path, record the environment, entry point, application boundary, dependencies, current telemetry, and escalation destination. Mark assumptions as coverage risks.

This inventory selects investigations to test. It does not establish what Sentry once observed. That is the first proof boundary.

Work one investigation end to end

Send a controlled request through a discovered user-visible path in a non-production environment. It should produce a known response or safe failure without personal data. Use a test identifier only if the application can preserve it under applicable data-handling rules.

The records below are illustrative and use symbolic values, not incident values:

EDGE EVENT
RayID=<ray-A> time=<time-A> host=<host-A> status=<status-A>
 
TRACE
trace_id=<trace-A> edge.ray_id=<ray-A> host=<host-A> route=<route-A>
 
PARSED LOG
trace_id=<trace-A> span_id=<span-A> service=<service-A> message=<message-A>
 
OWNER RESOLUTION
service=<service-A> owner=<owner-A> destination=<page-route-A>
first_action=<runbook-step-A>
 
MISSING-RayID BRANCH
search={time:<bounded-window>, host:<host-A>, route:<route-A>, status:<status-A>}
candidates=[<trace-B>, <trace-C>]
result=ambiguous; identity_not_proven=true

First locate the Cloudflare HTTP request event using RayID, a bounded timestamp range, and the request host. Confirm that the selected Logpush dataset supplies those fields. If the trace carries the same verified RayID, the edge-to-trace join is deterministic.

Next open a structured log emitted inside an active span. With a supported tracer and logging integration, injection enabled, and successful parsing, Datadog should expose trace_id and span_id. Test navigation from log to trace and back. Then resolve the normalized service to an accountable destination and a documented first action.

If RayID is absent from the trace, search with a narrow time window, host, normalized route, response status, and latency where available. This probabilistic fallback returns candidates. Retries and concurrent traffic can make the set ambiguous, so choosing one candidate would fabricate identity.

The records establish an observation: these links existed for this request under these conditions. Similar attributes without shared identity support an inference. Neither result proves that an edge condition caused application behavior; causal proof requires evidence from the failure itself.

Datadog states that logs and traces are sampled independently, so a retained log can contain an ID for a trace that was not ingested or retained. Its trace and log correlation documentation also requires JSON logs or correct parsing into attributes.

Treat correlation as a data contract

Control env, service, and version, along with normalized route templates, approved correlation identifiers, and ownership mappings. AWS resource tags can add raw context, but they do not establish agreement between infrastructure naming and application telemetry. Verify that agreement through an explicit mapping and an observed sample.

DD_LOGS_INJECTION=true does not decorate every log. Injection still depends on tracer and logger support; the logger must emit the properties, and the pipeline must parse them. Test records both inside and outside an active span using the language-specific instructions linked from Datadog’s documentation.

Cloudflare Logpush fields vary by dataset. For HTTP requests, select fields from the current schema and verify an ingested sample for type, meaning, null behavior, sensitivity, and cardinality. Do not mistake a response header for a Logpush field. Cloudflare’s Logpush documentation says delivery is not backfilled, making job health part of the investigation.

Keep raw paths, query values, credentials, bodies, customer identifiers, and other unbounded values out of tags. Retention cannot restore telemetry excluded by sampling or lost while delivery was disabled.

Derive monitors from operating conditions

Traffic rates, error budgets, delivery cadence, and historical baselines are unresolved, so fixed thresholds would be invented. Define the variables first:

W = evaluation window
D = expected maximum collector delivery delay
R_min = minimum request count accepted for a stable rate
E_window = allowable errors within W, derived from the error budget
G = consecutive breached evaluations required before notification
 
error_rate = errors(W) / requests(W)
 
Evaluate only when requests(W) >= R_min.
Alert when errors(W) > E_window for G consecutive evaluations.
Evaluate collection health when telemetry_age > D plus an agreed margin.

Derive R_min from observed volume and the owners’ accepted sample size. Measure D from collector and Logpush delivery cadence. Choose W and G against required response time and budget consumption.

Trace silence cannot distinguish an idle service from a stopped application, failed Agent, disabled integration, or broken pipeline. Pair symptom monitors with independent transport-health evidence such as Agent status, integration status, Logpush job health, or a policy-approved heartbeat. Healthy transport does not prove complete instrumentation.

Use a promotion review contract

Promote only when the tested investigation reaches an owner and first action without an unlabeled gap. Repeat it in later environments because permissions and configuration may differ.

The named failure mode is false correlation: unrelated edge and application events are joined, directing investigation toward the wrong service. The operating cost is maintaining mappings, retesting integrations after upgrades, controlling cardinality, monitoring delivery, and auditing ownership.

The unresolved proof boundary is coverage. One controlled request does not prove every route, low-frequency failure, sampled trace, or collector outage. It proves one navigable path under stated conditions and identifies the next path to test.

The next chapter applies the same discipline to privacy-safe production access logs, where emission and retrieval require separate verification.

FAQ

What if the service has no regular traffic?

Use a policy-approved synthetic request or heartbeat. If neither is allowed, no-data must remain informational because silence is ambiguous.

What if the edge identifier is not propagated?

Return the bounded candidate set and its matching attributes. Do not present one candidate as identified when concurrency or retries leave alternatives.

Why can a log ID lead to no trace?

Check parsing and identifier format, then ingestion and retention. Independent sampling can retain the log while excluding its associated trace.