mlsanityData to 7 Oct 2026

Speculative Speculative Decoding

arXiv 2603.03251 ↗First posted 3 Mar 2026cs.LGcited by 13

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Autoregressive decoding is bottlenecked by its sequential nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upcoming tokens from a slower target model, and then verifying them in parallel with a single target model forward pass. However, speculative decoding itself relies on a sequential dependence between speculation and verification. We introduce speculative speculative decoding (SSD) to parallelize these operations. While a verification is ongoing, the draft model predicts likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm. Our implementation is on average 30% faster than optimized speculative decoding baselines and up to 5x faster than autoregressive decoding with open source inference engines.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 46 references that are in the map. First posted on arXiv 3 Mar 2026.

Builds on

40 of 46 references.

Closest ancestors · 21

Same direction, newest first

  1. 2026DFlash: Block Diffusion for Flash Speculative Decoding5 Feb
  2. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference15 Oct
  3. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding28 Sep
  4. 2025SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding12 Jun
  5. 2025SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism16 May
  6. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  7. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification24 Feb
  8. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications7 Nov
  9. 2024AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration22 Oct
  10. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  11. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion10 Aug
  12. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
9 more
  1. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices4 Jun
  2. 2024Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference23 May
  3. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding3 Feb
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  8. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  9. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 19

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  8. 2025Large Language Diffusion Modelscited by 872
  9. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  12. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
7 more
  1. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  2. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  3. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  4. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  5. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  6. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  7. 2025Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Indexcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decodingcited by 7
  2. 2026See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMscited by 5
  3. 2026ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decodingcited by 3
  4. 2026A Practical Investigation of Training-free Relaxed Speculative Decodingcited by 2
  5. 2026K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modelingcited by 1
  6. 2026Adversarial Prompts for Acceptance Collapse in Speculative Decodingcited by 1
  7. 2026Cross-Family Speculative Decoding for Polish Language Models on Apple~Silicon: An Empirical Evaluation of Bielik~11B with UAG-Extended MLX-LMcited by 0
  8. 2026SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inferencecited by 0
  9. 2026Speculative Pipeline Decoding: Higher-Accuracy Drafting with Hidden Latency via Pipeline Parallelismcited by 0
  10. 2026Diffusion Language Model Parallel Decoding via Product-of-Experts Bridgecited by 0
  11. 2026AceSpec: An Asymmetric Edge-Cloud Collaborative Framework for Communication-Efficient LLM Inferencecited by 0
  12. 2026ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inferencecited by 0
1 more
  1. 2026AS2D: Accelerating On-Demand Audio Understanding on Mobile Devicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution
  2. 2026DLoop: Looped Speculative Decoding
  3. 2024SSSD: Simply-Scalable Speculative Decoding
  4. 2026LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
  5. 2026Approximate Speculative Decoding
  6. 2024A Theoretical Perspective for Speculative Decoding Algorithm
  7. 2024Block Verification Accelerates Speculative Decoding
  8. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding