Preserving the Real Client IP Through a Proxy Chain That Rewrites the Evidence

We needed the visitor’s address for regional routing, but every address reaching the application belonged to one of our proxies. An upstream hop had erased the only evidence of origin, so client IP provenance could not be recovered in application code.

Why client IP provenance collapsed

The request crossed a CDN, an intermediate proxy layer, Apache, nginx, and the application. Any layer could append, preserve, shorten, or overwrite forwarding headers. Syntax could show us a plausible address list, but it could not tell us which proxy observed an address or whether later hops protected that observation.

X-Forwarded-For often looks like a history:

203.0.113.44, 192.0.2.17, 192.0.2.28

That appearance is not evidence of how the history was built. Right-to-left parsing is justified only when every approved upstream hop appends its observed peer without reordering or deleting earlier entries. Under a dedicated-header contract, a designated provider instead overwrites a field from its transport observation, and controlled later hops preserve it. Configuration must select the provider’s documented behavior rather than infer it from a convincing-looking list.

Our failure mode was provenance collapse: an upstream overwrite discarded the original observation. Once that happened, no parser, alternate header name, or application fallback could reconstruct it.

The reusable formulation is: provenance is a value plus an observation point plus a protected path. The value is a normalized address. The observation point is the proxy that obtained it from a transport peer under a documented contract. The protected path prevents a client, bypass route, or later proxy from substituting another value before use.

The same reasoning applies to other proxy-derived security signals. A header claiming mutual-TLS identity, for example, is meaningful only if a known proxy performed the certificate check and the route to the consumer prevents direct header injection. The string alone carries no proof.

One auditable decision procedure

The first controlled ingress that still receives usable evidence should establish client IP provenance. The following table is both the configuration contract and the review artifact:

The upstream CIDRs in steps one through three identify peers allowed to attest forwarding data. TRUSTED_BACKEND_PEERS identifies machines allowed to deliver the already-derived value to the application. Those sets may overlap, but they answer different questions.

The ingress and backend checks share one normalizer. The ingress parser walks a provider-defined append chain from right to left; the backend accepts only the single canonical value delivered by an approved immediate peer:

from ipaddress import IPv6Address, ip_address, ip_network
 
TRUSTED_UPSTREAM_HOPS = tuple(
    ip_network(cidr) for cidr in ("192.0.2.0/28", "2001:db8:100::/64")
)
TRUSTED_BACKEND_PEERS = TRUSTED_UPSTREAM_HOPS
 
 
def normalized_ip(raw):
    if not isinstance(raw, str) or not raw.strip():
        return None
    try:
        address = ip_address(raw.strip())
    except ValueError:
        return None
    if isinstance(address, IPv6Address) and address.ipv4_mapped:
        return address.ipv4_mapped
    return address
 
 
def trusted(address, networks):
    return address is not None and any(address in network for network in networks)
 
 
def derive_from_append_chain(remote_addr, forwarded_for):
    peer = normalized_ip(remote_addr)
    if not trusted(peer, TRUSTED_UPSTREAM_HOPS):
        return None
    if not isinstance(forwarded_for, str) or not forwarded_for.strip():
        return None
 
    addresses = [normalized_ip(item) for item in forwarded_for.split(",")]
    if any(address is None for address in addresses):
        return None
 
    for candidate in reversed(addresses):
        if not trusted(candidate, TRUSTED_UPSTREAM_HOPS):
            return str(candidate)
    return None  # An all-trusted chain contains no defensible client boundary.
 
 
def canonical_client_ip(remote_addr, header_value):
    peer = normalized_ip(remote_addr)
    if not trusted(peer, TRUSTED_BACKEND_PEERS):
        return None
    if not isinstance(header_value, str) or "," in header_value:
        return None
    value = normalized_ip(header_value)
    return str(value) if value is not None else None

The CIDRs are documentation ranges and must be replaced with controlled values. Rejecting commas ensures that the canonical field remains one value rather than quietly becoming another forwarding chain.

Absence then becomes a product decision. A fail-closed policy can refuse any operation that requires trustworthy regional evidence. A degraded policy can continue with less precise or default regional routing while excluding the missing address from security decisions. The source does not establish which policy the system selected, so either behavior must be chosen explicitly, documented, and tested rather than attributed to this incident.

Operating cost and proof boundary

CIDR drift is the continuing operating cost. A missing provider range discards legitimate evidence; an overly broad range admits an unauthorized observation point. Range changes must be compared with authoritative network inventory, reviewed alongside the forwarding contract, and exercised through the deployed path. The source does not establish an automated update mechanism.

Origin bypass is the other material risk. Network policy should restrict backend listeners to approved immediate peers, while the application repeats the peer check independently. Alternate hostnames, routes, and health-proxy paths require coverage because a public route around ingress lets an attacker supply the canonical header directly.

A deployment test should capture, in a restricted diagnostic log, the ingress socket peer, received forwarding data, canonical output, backend socket peer, and final decision. Cases should cover injected canonical headers, malformed tokens, IPv4 and IPv6, multiple canonical values, unknown peers, stale or missing CIDRs, an all-trusted chain, and direct backend access. Provider documentation plus an end-to-end test must demonstrate the selected append, preserve, or overwrite behavior. Cloudflare’s original visitor IP guidance is one example of provider-specific header and origin guidance.

The observation is limited: an approved proxy presented a particular value. The operational inference is that the address approximates the originating network well enough for coarse regional behavior. It is not causal proof of a person, device, or original connection. NAT, VPNs, privacy relays, undocumented provider behavior, and evidence erased before controlled ingress remain outside the proof boundary.

FAQ

What if every address is trusted?

Return no address. The chain may be incomplete, the allowlist too broad, or the original observation absent.

What about IPv4-mapped IPv6 peers?

Normalize before CIDR comparison and define whether mapped addresses become IPv4. Otherwise listener configuration can change the trust decision.

Can PROXY protocol replace the header?

It can carry connection metadata across a controlled hop, but the receiver must accept it only from approved immediate peers. Public acceptance recreates the injection path.