Why Configured Telemetry Does Not Prove Production Tracing

The listener accepted the envelope, then failed before publishing it to the queue. When production operators went looking, they found no production tracing evidence showing where the work had vanished.

The application had four process types: a web process, queue workers, a scheduler, and a long-running listener. Existing integrations supplied boundaries for requests and scheduled jobs. The listener accepted envelopes, caught handler exceptions, and dispatched most work to a queue, but its command lifecycle did not produce a useful span for each delivery attempt.

The failure mode was an envelope-level telemetry gap. Exception capture covered some failures that reached the SDK, but did not explain successful, delayed, duplicated, or partially processed envelopes. The operating cost was incident ambiguity. An operator could not tell whether work stopped before publication, entered the queue, ran more than once, or completed without acknowledgement.

Review production tracing as evidence

OpenTelemetry separates telemetry generation from the systems that receive and analyze it. Its observability primer explains signals and context propagation, while the instrumentation documentation covers the code and agents that generate telemetry.

Use this evidence ladder once during review, recording the strongest level actually demonstrated:

Configuration supports an inference that telemetry was intended. Seeing completed spans is a runtime observation. Causal proof requires evidence connecting the relevant transitions while excluding credible alternatives. None of these levels proves that the transitions were atomic or that the application behaved correctly.

For the listener, one envelope delivery attempt is the useful trace boundary, not the lifetime of a process that may run for days. A later queue delivery gets its own consumer span. Context should be carried in allowlisted metadata rather than the business payload, and untrusted inbound trace identifiers should not automatically become parents.

Publication has an indeterminate state

Queue publication and envelope acknowledgement are separate commits. Even the publication result is not always knowable: a timeout may mean the queue rejected the message, or that it accepted the message but its confirmation was lost. Treating every timeout as failure can create a duplicate on retry. Treating it as success can lose work if the queue rejected it.

The worker has the same shape of uncertainty. An external side effect can complete before processing throws, leaving the worker unable to report whether retrying is safe. A later acknowledgement failure can also cause completed work to be delivered again.

A trace can record the attempted transitions, timestamps, exceptions, and confirmations that were observed. It cannot recover a lost queue confirmation, establish whether an external side effect committed, or atomically coordinate the listener, broker, worker, and side-effect system.

Containing that ambiguity requires an application guarantee. A stable idempotency key can make retries safe at the side-effect boundary. Durable deduplication can suppress repeated logical work. A transactional outbox can commit an intent with local state before reliable publication; an inbox can durably record consumed identifiers before processing. Where duplicates are acceptable, the contract can instead state accepted at-least-once semantics. The source did not establish which guarantee existed, so duplicate-safe publication and processing remain the unresolved proof boundary.

Record outcomes without inventing certainty

This language-neutral pseudocode uses three publication and side-effect outcomes. Placeholder APIs and attributes must be adapted to the deployed telemetry version.

enum Outcome { CONFIRMED, FAILED, UNKNOWN }
 
async publishAttempt(job, metadata):
    span = startProducerSpan(currentContext())
    outcome = UNKNOWN
    try:
        with makeCurrent(span):
            injectTraceContext(currentContext(), metadata)
            confirmation = await queue.publish(job.payload, metadata)
            if confirmation.accepted:
                outcome = CONFIRMED
                span.setStatus(OK)
            else:
                outcome = FAILED
                span.setStatus(ERROR)
    catch error:
        # A timeout or broken connection does not prove rejection.
        outcome = UNKNOWN if confirmationMayBeLost(error) else FAILED
        span.recordException(sanitize(error))
        span.setStatus(ERROR)
    finally:
        span.setAttribute("publish.outcome", boundedName(outcome))
        span.end()
    return outcome
 
async onEnvelope(envelope, runtime):
    consumer = startConsumerSpan(extractTrustedContext(envelope.metadata))
    publishOutcome = UNKNOWN
    try:
        with makeCurrent(consumer):
            job = await handle(envelope.payload)
            metadata = copyAllowlistedMetadata(job.metadata)
            metadata["idempotency_key"] = stableApprovedKey(envelope)
            publishOutcome = await publishAttempt(job, metadata)
 
            if publishOutcome == CONFIRMED:
                await envelope.acknowledge()
                consumer.setStatus(OK)
            elif publishOutcome == FAILED:
                await envelope.reject(requeue = retryPolicyAllows())
                consumer.setStatus(ERROR)
            else:
                await resolveUnknownPublication(envelope, metadata)
                consumer.setStatus(ERROR)
    finally:
        consumer.setAttribute("publish.outcome", boundedName(publishOutcome))
        consumer.end()
        if runtime.isTerminating(): await telemetry.forceFlush()
 
async onWorkerMessage(message, runtime):
    worker = startConsumerSpan(extractTrustedContext(message.metadata))
    sideEffectOutcome = UNKNOWN
    try:
        with makeCurrent(worker):
            result = await processWithIdempotencyKey(
                message.payload, message.metadata["idempotency_key"])
            sideEffectOutcome = result.confirmed ? CONFIRMED : UNKNOWN
            await message.acknowledge()
            worker.setStatus(OK)
    catch error:
        # Processing may throw after the side effect commits.
        sideEffectOutcome = classifyWithoutGuessing(error, sideEffectOutcome)
        worker.recordException(sanitize(error))
        worker.setStatus(ERROR)
        await message.reject(requeue = retryPolicyAllows(error))
    finally:
        worker.setAttribute("side_effect.outcome", boundedName(sideEffectOutcome))
        worker.end()
        if runtime.isTerminating(): await telemetry.forceFlush()

stableApprovedKey, processWithIdempotencyKey, and resolveUnknownPublication are requirements, not evidence that the audited application implemented them. Each redelivery should create a distinct consumer span. Use bounded attempt numbers, a redelivery Boolean, and a small outcome vocabulary. Do not attach payloads, addresses, user-derived queue names, raw exception bodies, or proprietary identifiers.

Test from the operator’s starting point

Run one approved representative event and record its time window, bounded correlation value, listener outcome, publication result, worker outcome, exporter diagnostics, backend query, and notification result. Retrieval must start with facts an operator would possess during an incident, not a trace URL supplied by the tester.

Sampling limits the conclusion. Head sampling can discard a trace before a later failure is known, so one missing trace does not prove exporter failure. Tail sampling can retain traces using completed outcomes, but adds buffering, decision latency, infrastructure cost, and dependence on sufficiently complete trace arrival. Any temporary sampling rule needs a defined scope, owner, duration, and removal plan.

The source review established instrumentation intent and found the missing envelope boundary. It did not establish completed production spans, successful export, ingestion, incident-time retrieval, notification delivery, or duplicate-safe processing. Those need controlled runtime and correctness tests.

The migration chapter shows how the same distinction between configured and retrievable evidence shapes an end-to-end observability investigation.

FAQ

Does exception capture prove successful deliveries are traced?

No. It observes an error-reporting path. Successful work, incomplete spans, and commit ambiguity require separate evidence.

Can tracing prevent duplicate processing?

No. It can expose repeated attempts. Safe retry requires idempotency, deduplication, transactional coordination, or accepted at-least-once semantics.

What if synthetic production events are forbidden?

Observe a pre-approved real event without replaying or altering its payload. Record which retry, sampling, unknown-outcome, retrieval, and notification branches remain untested.