# Speeding up Speculative Decoding via Sequential Approximate Verification

arXiv: https://arxiv.org/abs/2502.04557 · HTML: https://mlsanity.com/paper/2502.04557/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 22 of 22)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [Speculative Decoding with Big Little Decoder](https://mlsanity.com/paper/2302.07863/) (2023)
- [SpecTr: Fast Speculative Decoding via Optimal Transport](https://mlsanity.com/paper/2310.15141/) (2023)
- [SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths](https://mlsanity.com/paper/2405.19715/) (2024)
- [Learning to Decode Collaboratively with Multiple Language Models](https://mlsanity.com/paper/2403.03870/) (2024)
- [Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment](https://mlsanity.com/paper/2501.19309/) (2025)
- [Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion](https://mlsanity.com/paper/2408.05636/) (2024)
- [Transfer Q Star: Principled Decoding for LLM Alignment](https://mlsanity.com/paper/2405.20495/) (2024)
- [Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models](https://mlsanity.com/paper/2405.04304/) (2024)
- [A Theoretical Perspective for Speculative Decoding Algorithm](https://mlsanity.com/paper/2411.00841/) (2024)
- [AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probability](https://mlsanity.com/paper/2410.18351/) (2024)
- [From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice](https://mlsanity.com/paper/2410.01812/) (2024)
- [Improving Multi-candidate Speculative Decoding](https://mlsanity.com/paper/2409.10644/) (2024)
- [Language Model Sentence Completion with a Parser-Driven Rhetorical Control Method](https://mlsanity.com/paper/2402.06125/) (2024)
- [Approximately Aligned Decoding](https://mlsanity.com/paper/2410.01103/) (2024)

## Built on by (most cited first, 11 of 11)

- [Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match](https://mlsanity.com/paper/2511.22972/) (2025)
- [Parallel Token Prediction for Language Models](https://mlsanity.com/paper/2512.21323/) (2025)
- [See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs](https://mlsanity.com/paper/2604.05650/) (2026)
- [When, What, and How: Rethinking Retrieval-Enhanced Speculative Decoding](https://mlsanity.com/paper/2511.01282/) (2025)
- [DSDE: Dynamic Speculative Decoding with KLD Stability for Real-World Serving](https://mlsanity.com/paper/2509.01083/) (2025)
- [Beyond the Target: From Imitation to Collaboration in Speculative Decoding](https://mlsanity.com/paper/2605.24793/) (2026)
- [Accelerating Speculative Diffusions via Block Verification](https://mlsanity.com/paper/2606.13426/) (2026)
- [Prompt Fairness: Sub-group Disparities in LLMs](https://mlsanity.com/paper/2511.19956/) (2025)
- [Annealed Relaxation of Speculative Decoding for Faster Autoregressive Image Generation](https://mlsanity.com/paper/2601.09212/) (2026)
- [CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration](https://mlsanity.com/paper/2605.11186/) (2026)
- [Self-Speculation for Faster Reasoning Models](https://mlsanity.com/paper/2608.20359/) (2026)

## Abstract (arXiv)

Speculative Decoding (SD) is a recently proposed technique for faster inference using Large Language Models (LLMs). SD operates by using a smaller draft LLM for autoregressively generating a sequence of tokens and a larger target LLM for parallel verification to ensure statistical consistency. However, periodic parallel calls to the target LLM for verification prevent SD from achieving even lower latencies. We propose SPRINTER, which utilizes a low-complexity verifier trained to predict if tokens generated from a draft LLM would be accepted by the target LLM. By performing sequential approximate verification, SPRINTER does not require verification by the target LLM and is only invoked when a token is deemed unacceptable. This reduces the number of calls to the larger LLM, achieving further speedups and lower computation cost. We present a theoretical analysis of SPRINTER, examining the statistical properties of the generated tokens, as well as the expected reduction in latency as a function of the verifier. We evaluate SPRINTER on several datasets and model pairs, demonstrating that approximate verification can still maintain high quality generation while further reducing latency.