Cross-Datacenter PD Disaggregation: How PDD Cut LLM Inference Latency in Half

At the close of the 2026 WAIC conference in late July, Infinigence-AI released the full technical report for PDD (Prefill-RelayDecode-MainDecode), a new cross-datacenter heterogeneous inference architecture. On DeepSeek-V4-pro with real production traces, the system delivered a 46% reduction in P90 Time-to-First-Token (TTFT) and a 37.5% improvement in cost-efficiency — a practical answer to a problem that has crippled cross-datacenter LLM serving for years: how to move massive KV cache payloads over wide-area Ethernet without ruining user experience.

The Long-Standing "Wall of Sighs" in Cross-Datacenter PD

LLM inference has two distinct phases: Prefill (compute-bound) and Decode (memory-bandwidth-bound). The textbook ideal is heterogeneous disaggregation — send Prefill to compute-heavy GPUs, send Decode to memory-bandwidth-heavy accelerators. But building a single intra-datacenter cluster with a balanced mix of both is prohibitively expensive, and rigid hardware ratios become obsolete the moment the model architecture changes.

The natural alternative is to connect geographically distributed, already-deployed homogeneous clusters over low-cost wide-area Ethernet, and run heterogeneous inference across them. The idea is elegant. The engineering reality is brutal: cross-cluster KV cache transfer is starved of bandwidth and dominated by latency, and neither compression nor higher link rates can hide the round-trip cost once a request is in flight.

Three Insights Before the Breakthrough

Before designing PDD, the Infinigence team surfaced three insights in the report's Background section. These observations are the foundation that makes the whole architecture possible.

Insight 1: Hardware is "extremely biased" between phases. Internal benchmarks on a baseline of 8 high-end GPUs revealed dramatic asymmetry. One vendor's "E" chip hit only 81% of baseline performance in the Prefill phase but blew past 210% in Decode. Inside a homogeneous cluster, that kind of specialization becomes a liability instead of an asset.

Insight 2: Decode-side RadixCache (DRC) cuts cross-cluster bandwidth by 10×. Agentic workloads are characterized by extremely high prefix-cache hit rates. By caching historical KV cache on decode instances and only transmitting the cache delta for hit requests, DRC reduces required wide-area bandwidth by up to 10× at a 90% average hit rate.

Insight 3: Hit-rate distribution is "extremely skewed" — not uniform. Analysis of 6 million production traces revealed that 70%–80% of requests sit at hit rates above 95%, while only a small minority carry giant KV cache payloads. Microbenchmarks on a 20 Gbps wide-area link showed P50 transfer latency of just 248 ms under the real skewed distribution. Under a hypothetical uniform distribution, however, the connection pool saturates immediately and P50 latency jumps to 1210 ms — plus 1682 ms of queueing on top.

Insight 3 is the one that unlocks the design: extreme tail latency is concentrated in a tiny fraction of "problem requests." The latency problem can be solved locally for that fraction alone — no need to globally optimize the network.

The Three-Tier Relay Architecture

PDD inserts a middle layer — RelayDecode (RLD) — between the conventional Prefill (P) and Decode (D) tiers:

• P instance: Lives in the primary cluster, processes prompts, generates KV cache.
• RLD instance: Co-located with P in the same datacenter, connected via high-speed RDMA (transfer latency: tens of milliseconds).
• MD instance (MainDecode): Deployed in a distant cluster, connected over wide-area Ethernet (transfer latency: seconds to tens of seconds).

The execution model is a 2×100-meter relay race. The moment P finishes Prefill, it does two things in parallel: ships the KV cache to the same-room RLD over RDMA, and ships it to the faraway MD over Ethernet. RLD starts decoding and streaming tokens to the user immediately — masking the much slower cross-cluster transfer. Several seconds later, when MD has the complete KV cache, decoding seamlessly hands off from RLD to MD, and MD produces the bulk of the output.

What the user perceives is the tens-of-milliseconds RLD latency over RDMA. They never see the wide-area transfer at all. PDD uses a small amount of local compute (RLD) to surgically mask the long tail caused by a small fraction of low-hit-rate requests.

Extend-Decode Handoff: Transfer Token IDs, Recompute KV Cache Locally

The hardest engineering challenge is the handoff itself. Once MD is ready, how does it take over decoding from RLD without introducing a new round of network pain?

The naive approach is to incrementally stream RLD's freshly generated KV cache to MD over Ethernet. But that re-introduces the very latency the architecture is trying to hide — and adds H2D/D2H copy overhead on top.

PDD's answer is counter-intuitive but elegant: RLD never transmits KV cache during handoff. It only sends lightweight CPU-resident Token IDs. Each handoff payload is just a few kilobytes, regardless of how much KV cache the tokens represent. MD then uses an Extend-Decode technique (the same family of tricks behind Multi-Token Prediction and speculative decoding) to recompute the corresponding KV cache locally while generating the next token.

Is local recomputation slow? Not really. Decode is memory-bandwidth-bound, not compute-bound. The team measured average recomputation time of 305.2 ms for 100 tokens, with a maximum of 321.4 ms and a standard deviation of just 4.8 ms. By contrast, naive Ethernet KV transfer averaged 185.4 ms but spiked to 512.6 ms under congestion with a 96.3 ms standard deviation, plus an extra 24.5 ms of H2D/D2H overhead. PDD turns a network-flaky, unpredictable operation into a deterministic local computation.

Two handoff modes — Catch-Up Handoff and One-Shot Handoff — can be alternated dynamically based on RLD load. Catch-Up preserves user-facing OTPS smoothness; One-Shot minimizes RLD occupation at the cost of a brief throughput dip.

Pipeline Orchestration: Keep RLD Thin

RLD is precious compute. The design principle is simple: RLD only exists to mask latency, never to do real work. The team built two pipelines based on the skewed hit-rate insight:

• P-MD pipeline: High-hit-rate requests (hit rate > 95%, 70%+ of traffic) have tiny KV cache payloads, so they bypass RLD entirely and go straight to MD.
• P-RLD-MD pipeline: Only low-hit-rate outliers — the "problem requests" — get routed through RLD for latency masking.

In production measurements, RLD handled just 6.2% of total output tokens (about 270,000 tokens), while MD carried the remaining 93.8% (about 4.12 million) — a 15.2× load imbalance by design. The architecture also guarantees that MD's cache hit rate stays aligned with P's, which keeps RLD's load stable and prevents a vicious positive-feedback loop that would otherwise arise if requests lived and died at RLD.

Production Scorecard: A 37.5% BCR Lift

Architecture only matters if it survives real traffic. The team benchmarked PDD on DeepSeek-V4-pro with real agentic workload traces, against two baselines: CDC-PD (cross-datacenter heterogeneous PD without RLD masking) and IDC-PD (intra-datacenter homogeneous PD on a single high-end GPU cluster).

Latency. CDC-PD's P90 TTFT hit 18.3 seconds and P99 approached 30 seconds. PDD dropped P90 TTFT to 9.8 seconds (−46%) and P99 to 14.4 seconds — comfortably under a 10 s / 15 s SLA envelope.

Load distribution. RLD: 6.2% of output tokens. MD: 93.8%. The ratio is by design, and it stayed there.

Cost-efficiency. Under strict SLA constraints (P90 TTFT < 10 s, P99 < 15 s), PDD delivered 27.8% higher RPS Goodput at 3.5%–7.1% lower total cost than IDC-PD, producing a 37.5% improvement in Benefit-Cost Ratio (BCR).

That isn't the ceiling either. Test resources limited RLD to just three GPU-A nodes (the minimum DP-parallel size to run DeepSeek-V4-pro). At larger deployments, RLD's fixed overhead would be amortized further; pairing it with more aggressive inference-specialized chips would widen the gap. And MTP-style optimizations were disabled in the test because of temporary incompatibility with DRC — turning them on should push the BCR higher still.

What This Means for Next-Generation MaaS Infrastructure

PDD is not just a trick for hiding wide-area Ethernet latency. It encodes Infinigence's design philosophy for the next generation of Model-as-a-Service (MaaS) infrastructure: minimal local heterogeneity plus flexible remote disaggregation. Datacenters no longer need to assemble elaborate heterogeneous hardware stacks under one roof. They can keep a small pool of compute-heavy chips for Prefill in the primary site, and let distant homogeneous clusters handle the bulk of memory-bound decoding. The economics of the inference supply chain are being redrawn.

The team has open-sourced the full PDD technical report on GitHub (see infinigence/pdd) and committed to updating it with future progress. In 2026, when inference cost remains the industry's "make-or-break line," this relay team may be the key that unlocks the next phase of scale-out.

Sources:

1. QbitAI, "Cross-Datacenter 'Relay Run' Activates National Compute: PD Disaggregation Finally Breaks Through — Latency Halved, Cost Down Nearly 40%," 2026-07-30.
2. Infinigence-AI, PDD: Unleashing Economical and Flexible Heterogeneous LLM Inference via Cross-Datacenter Prefill-Decode Disaggregation (technical report), 2026-07.
3. GitHub: infinigence/pdd.