Skip to main content

JetSpec: Breaking the Scaling Ceiling of Speculative Decoding

Introduction: The Serial Bottleneck of LLM Decoding​

Modern autoregressive LLMs rely on sequential token generation, which becomes the dominant latency cost in math, coding, and agentic reasoning workloads. Speculative decoding (SD) accelerates this by drafting multiple tokens and verifying them in parallel, and has become the de facto low-cost inference technique across the LLM serving stack.

The Causality-Efficiency Dilemma​

SD has a fundamental scaling ceiling: larger draft budgets translate into real speedup only when acceptance stays high and drafting overhead stays low. Existing head-based methods (which attach a lightweight draft head to the target model) face a causality-efficiency dilemma:

  • Autoregressive drafters such as EAGLE produce high-quality path-conditioned candidates, but their drafting cost grows with tree depth.
  • Bidirectional block-diffusion drafters such as DFlash generate all positions in one forward pass, but their branch-agnostic marginals can form individually plausible yet mutually inconsistent trees, wasting budget and reducing acceptance.

Neither approach lets practitioners simply "throw more budget at the problem".

JetSpec's Core Innovation​

In June 2026, StepFun, together with UC San Diego, Zhejiang University, UIUC, and Nanjing University, released JetSpec (arXiv:2606.18394v3) to break this dilemma. Three design pillars make it work:

  1. Causal parallel draft head: a single head that predicts multiple tree nodes in one forward pass.
  2. Block-level causal attention: applied over fused hidden states drawn from the frozen target model, preserving branch-level causal conditioning.
  3. Path-conditioned alignment: each branch's draft distribution aligns with the target model's autoregressive factorization, so candidate tree scores genuinely reflect the target's path-conditioned probabilities.

The net effect: larger draft budgets stably convert into longer accepted prefixes and higher end-to-end speedup.

Experimental Results: 9.64x on H100​

JetSpec was systematically evaluated on math, coding, and chat benchmarks against DFlash, DDTree (DFlash's tree variant), and the EAGLE family, on Qwen3 dense and MoE models:

  • H100, MATH-500, peak speedup: 9.64x.
  • H100, MATH-500, 256-token high budget: mean acceptance length τ=10.7, end-to-end 9.5x speedup.
  • H100, MT-Bench, high budget: τ=5.9, over 4x speedup.
  • Coding and math workloads: 1000+ TPS (tokens per second), acceptance length up to 10x.

At low budgets (16 tokens) JetSpec is competitive with baselines; at high budgets (256 tokens) its lead widens dramatically—clear evidence that it converts draft budget into accepted length.

Industrial Integration and Engineering​

  • Native vLLM integration detailed in Appendix E of the paper.
  • Evaluated across request rates on both H100 and B200 GPUs.
  • Beats DFlash, DDTree, and EAGLE-family baselines consistently under small to moderate serving loads.
  • Stronger gains on higher-end GPUs such as B200, hinting at meaningful algorithm–hardware co-design headroom.
  • Distillation training cost is sub-1B tokens, keeping deployment overhead controllable.

Industry Impact​

Shortly after DeepSeek introduced DSpark (claiming up to 85% per-user TPS gains), StepFun released JetSpec, signaling that the LLM inference efficiency race has entered a new "algorithm + hardware co-optimization" phase. For Agent-style high-frequency, long-chain workloads, JetSpec cuts per-step decoding latency by roughly an order of magnitude—a major 2026 milestone in the LLM serving stack.

Code and models are open-sourced at https://github.com/hao-ai-lab/JetSpec