Speculative Speculative Decoding: How Saguaro Cuts LLM Draft Latency to Zero

Speculative decoding (SD) transformed large language model inference by using a compact draft model to propose tokens that a massive target model verifies in a single forward pass. Yet, standard speculative decoding harbors a fundamental flaw: it is strictly synchronous. While the target model verifies, the draft model idles; while the draft model speculates, the target model waits.

A team of researchers—led by Tanishq Kumar, Tri Dao, and Avner May—have broken this sequential barrier with Speculative Speculative Decoding (SSD) and its reference algorithm, Saguaro. By disaggregating drafting and verification onto separate hardware and anticipating likely verification outcomes in parallel, Saguaro eliminates draft-phase latency entirely, unlocking a 30% speedup over state-of-the-art speculative engines (including SGLang and vLLM baselines) and up to a 5x speedup over standard autoregressive decoding on Llama-3.1-70B and Qwen-3 architectures.

---

The Synchronous Wall in Speculative Decoding

Standard speculative decoding relies on a basic two-phase loop:

1. A fast draft model M_draft autoregressively generates a speculative lookahead sequence of K candidate tokens: s = (s₁, …, s_K).
2. The high-capacity target model M_target computes logits across the proposed prefix in one parallel step, accepting tokens sequentially until the first divergence, then sampling an exact "bonus token" t* from the residual distribution r(t) ∝ max(p_target(t) - p_draft(t), 0).

Even with highly optimized kernels, the GPU cluster hosting the target model remains completely idle while waiting for the draft model to produce its next guess.

Standard Speculative Decoding (Synchronous):
[ Draft 1..K ] ──> [ Verify 1..K ] ──> [ Draft 1..K ] ──> [ Verify 1..K ]
                     (Draft Idles)       (Target Idles)

Speculative Speculative Decoding (SSD / Asynchronous):
GPU 0-3 (Target): ──[ Verify Round T-1 ]─────────────[ Verify Round T ]──
                           ▲                                ▲
                           │ (instant cache hit)            │
GPU 4 (Draft):    ──[ Speculate Cache for T ]────────[ Speculate Cache for T+1 ]──

SSD completely decouples these roles. The target verifier runs across tensor-parallel GPUs (e.g., 4× NVIDIA H100s or AMD Instinct MI300X accelerators), while the draft model runs concurrently on a dedicated accelerator (e.g., 1× H100). While the verifier executes verification for round T-1, the draft model actively constructs a speculation cache 𝒮ᵀ containing candidate continuations for multiple prospective verification outcomes. If the verifier's actual result hits the cache, the next sequence of drafted tokens is fed to the verifier instantly with zero drafting overhead.

---

The O(KV) Problem: Taming Combinatorial Explosion

A speculative verification outcome is defined as v = (k, t*), where k ∈ [0, K] represents the number of accepted tokens, and t* is the bonus token sampled from the vocabulary V. Because there are (K + 1)V possible outcomes, a draft model cannot exhaustively pre-compute every branch.

To make asynchronous speculative decoding tractable within the time it takes the target model to execute a single forward pass, Saguaro introduces three core theoretical breakthroughs:

1. Geometric Fan-Out Cache Construction

Rather than distributing speculative compute uniformly across all sequence positions, Saguaro proves that verification string lengths follow a capped geometric distribution governed by the base acceptance rate αₚ and an empirical power-law rejection rate r.

Saguaro calculates the optimal allocation of bonus token guesses (fan-out Fₖ) for each lookahead position k under a computational budget constraint Σₖ₌₀^K Fₖ ≤ B:

Fₖ = F₀ · αₚ^(k / (1 + r)) (∀ k < K)

F_K = F₀ · αₚ^(K / (1 + r)) · (1 - αₚ)^(-1 / (1 + r))

By allocating more fan-out budget to earlier token positions where rejections are statistically most likely, Saguaro maximizes the cache hit probability pₕᵢₜ without wasting compute on highly improbable long-tail prefixes.

2. Saguaro Sampling: Biasing the Residual Distribution

Predicting the exact bonus token t* is notoriously difficult because it is sampled from the target's residual distribution. Saguaro turns this mathematical relationship on its head.

Because r(t) ∝ max(p_target(t) - p_draft(t), 0), decreasing the draft probability p_draft(t) on a token directly increases that token's probability mass inside the residual distribution. Saguaro introduces a cache-aware sampling function σ_(F, C)(z) with downweighting constant C ∈ [0, 1]:

σ_(F, C)(z)ₜ ∝ C · exp(zₜ) if t ∈ top_F(z) ; exp(zₜ) otherwise

By intentionally suppressing draft logits on the top-F cached tokens, Saguaro forces the verifier's residual distribution to concentrate on those exact cached tokens by construction.

3. Adaptive Batch Fallback

At higher batch sizes, the probability of at least one batch element suffering a cache miss approaches 1. If every cache miss triggered a slow just-in-time neural speculation, the entire batch would stall.

Saguaro solves this with an adaptive threshold b*: for batch sizes b < b*, it uses full just-in-time neural drafting as a fallback; for b ≥ b*, it falls back to low-latency non-neural (or random/n-gram) speculative tokens.

---

Benchmark Results and Real-World Impact

In extensive evaluations across mathematical reasoning (GSM8K), code generation (HumanEval), and instruction following (Alpaca, UltraFeedback):

• Latency Reduction: Serving Llama-3.1-70B with a Llama-3.2-1B draft model achieved up to 5x faster generation compared to standard autoregressive generation.
• Superseding SOTA Speculative Engines: Saguaro outperformed optimized SGLang and vLLM speculative setups by an average of 30% end-to-end latency reduction across all test distributions.
• Pareto Frontier Dominance: Unlike prior speculative decoding schemes that trade massive amounts of GPU compute for modest latency gains, SSD strictly improves the throughput-latency Pareto curve by saturating previously idle draft hardware.
• Cross-Architecture Verification: Recent independent reproductions by AMD on Instinct MI300X clusters (ROCm 7.2) confirmed identical scaling laws and verified SSD's compatibility with FlashInfer and custom grouped-query attention masks.

---

The Future of Disaggregated Inference

As reasoning-heavy models (such as deep chain-of-thought architectures) demand hundreds of output tokens per query, inference costs are dominated by generation latency. Speculative Speculative Decoding marks a critical paradigm shift: viewing inference not as a rigid sequential pipeline, but as an asynchronous speculative execution graph.

By treating draft speculation as a pre-computed cache of forking paths, Saguaro proves that the time spent verifying one token can be seamlessly leveraged to draft the next ten—paving the way for cluster-level disaggregated speculative decoding at scale.