# Speculative Decoding and Beyond: An In-Depth Survey of Techniques

arXiv: https://arxiv.org/abs/2502.19732 · HTML: https://mlsanity.com/paper/2502.19732/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 99 of its 113 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 30 of 99)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [Introduction to Multi-Armed Bandits](https://mlsanity.com/paper/1904.07272/) (2019)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [Pay Attention to MLPs](https://mlsanity.com/paper/2105.08050/) (2021)
- [Better & Faster Large Language Models via Multi-token Prediction](https://mlsanity.com/paper/2404.19737/) (2024)
- [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018)
- [LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding](https://mlsanity.com/paper/2404.16710/) (2024)
- [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation](https://mlsanity.com/paper/2203.16487/) (2022)
- [DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://mlsanity.com/paper/2310.08461/) (2023)
- [Speculative Decoding with Big Little Decoder](https://mlsanity.com/paper/2302.07863/) (2023)
- [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://mlsanity.com/paper/2304.04675/) (2023)

## Built on by (most cited first, 23 of 23)

- [SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding](https://mlsanity.com/paper/2603.18567/) (2026)
- [DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding](https://mlsanity.com/paper/2505.19201/) (2025)
- [KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimization](https://mlsanity.com/paper/2505.16162/) (2025)
- [Draft-based Approximate Inference for LLMs](https://mlsanity.com/paper/2506.08373/) (2025)
- [Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing](https://mlsanity.com/paper/2510.16040/) (2025)
- [Steering Pretrained Drafters during Speculative Decoding](https://mlsanity.com/paper/2511.09844/) (2025)
- [Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs](https://mlsanity.com/paper/2602.15318/) (2026)
- [Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling](https://mlsanity.com/paper/2604.04987/) (2026)
- [FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning](https://mlsanity.com/paper/2509.00195/) (2025)
- [Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios](https://mlsanity.com/paper/2511.20340/) (2025)
- [Accelerated Test-Time Scaling with Model-Free Speculative Sampling](https://mlsanity.com/paper/2506.04708/) (2025)
- [P/D-Device: Disaggregated Large Language Model between Cloud and Devices](https://mlsanity.com/paper/2508.09035/) (2025)
- [TABED: Test-Time Adaptive Ensemble Drafting for Robust Speculative Decoding in LVLMs](https://mlsanity.com/paper/2601.20357/) (2026)
- [Efficient Epistemic Uncertainty Estimation for Large Language Models via Knowledge Distillation](https://mlsanity.com/paper/2602.01956/) (2026)
- [Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding](https://mlsanity.com/paper/2605.14005/) (2026)
- [SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding](https://mlsanity.com/paper/2606.00021/) (2026)
- [DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation](https://mlsanity.com/paper/2606.00535/) (2026)
- [ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping](https://mlsanity.com/paper/2608.24411/) (2026)
- [Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices](https://mlsanity.com/paper/2602.08060/) (2026)
- [Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generation](https://mlsanity.com/paper/2603.18428/) (2026)
- [DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Execution](https://mlsanity.com/paper/2605.28678/) (2026)
- [HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworks](https://mlsanity.com/paper/2609.36357/) (2026)
- [CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters](https://mlsanity.com/paper/2610.00321/) (2026)

## Abstract (arXiv)

Sequential dependencies present a fundamental bottleneck in deploying large-scale autoregressive models, particularly for real-time applications. While traditional optimization approaches like pruning and quantization often compromise model quality, recent advances in generation-refinement frameworks demonstrate that this trade-off can be significantly mitigated. This survey presents a comprehensive taxonomy of generation-refinement frameworks, analyzing methods across autoregressive sequence tasks. We categorize methods based on their generation strategies (from simple n-gram prediction to sophisticated draft models) and refinement mechanisms (including single-pass verification and iterative approaches). Through systematic analysis of both algorithmic innovations and system-level implementations, we examine deployment strategies across computing environments and explore applications spanning text, images, and speech generation. This systematic examination of both theoretical frameworks and practical implementations provides a foundation for future research in efficient autoregressive decoding.