# Block Verification Accelerates Speculative Decoding

arXiv: https://arxiv.org/abs/2403.10444 · HTML: https://mlsanity.com/paper/2403.10444/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 17 of 17)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://mlsanity.com/paper/2310.08461/) (2023)
- [REST: Retrieval-Based Speculative Decoding](https://mlsanity.com/paper/2311.08252/) (2023)
- [Cascade Speculative Drafting for Even Faster LLM Inference](https://mlsanity.com/paper/2312.11462/) (2023)
- [Inference with Reference: Lossless Acceleration of Large Language Models](https://mlsanity.com/paper/2304.04487/) (2023)

## Built on by (most cited first, 30 of 30)

- [DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding](https://mlsanity.com/paper/2510.02358/) (2025)
- [Speculative Decoding and Beyond: An In-Depth Survey of Techniques](https://mlsanity.com/paper/2502.19732/) (2025)
- [BanditSpec: Adaptive Speculative Decoding via Bandit Algorithms](https://mlsanity.com/paper/2505.15141/) (2025)
- [Towards Optimal Multi-draft Speculative Decoding](https://mlsanity.com/paper/2502.18779/) (2025)
- [TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees](https://mlsanity.com/paper/2601.07353/) (2026)
- [Guided Speculative Inference for Efficient Test-Time Alignment of LLMs](https://mlsanity.com/paper/2506.04118/) (2025)
- [Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference](https://mlsanity.com/paper/2405.14105/) (2024)
- [Traversal Verification for Speculative Tree Decoding](https://mlsanity.com/paper/2505.12398/) (2025)
- [A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models](https://mlsanity.com/paper/2405.13019/) (2024)
- [SSR: Speculative Parallel Scaling Reasoning in Test-time](https://mlsanity.com/paper/2505.15340/) (2025)
- [Speculative Coupled Decoding for Training-Free Lossless Acceleration of Autoregressive Visual Generation](https://mlsanity.com/paper/2510.24211/) (2025)
- [Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding](https://mlsanity.com/paper/2601.05724/) (2026)
- [A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challenges](https://mlsanity.com/paper/2602.05883/) (2026)
- [Trees from Marginals: Autoregressive drafting with factorized priors](https://mlsanity.com/paper/2607.06763/) (2026)
- [A Practical Investigation of Training-free Relaxed Speculative Decoding](https://mlsanity.com/paper/2607.08690/) (2026)
- [Out-of-Vocabulary Sampling Boosts Speculative Decoding](https://mlsanity.com/paper/2506.03206/) (2025)
- [Global Resolution: Optimal Multi-Draft Speculative Sampling via Convex Minimization](https://mlsanity.com/paper/2511.15898/) (2025)
- [Beyond the Target: From Imitation to Collaboration in Speculative Decoding](https://mlsanity.com/paper/2605.24793/) (2026)
- [Accelerating Speculative Diffusions via Block Verification](https://mlsanity.com/paper/2606.13426/) (2026)
- [Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding](https://mlsanity.com/paper/2509.19368/) (2025)
- [Adversarial Prompts for Acceptance Collapse in Speculative Decoding](https://mlsanity.com/paper/2607.21804/) (2026)
- [Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding](https://mlsanity.com/paper/2609.02897/) (2026)
- [Fast Inference via Hierarchical Speculative Decoding](https://mlsanity.com/paper/2510.19705/) (2025)
- [Greedy Multi-Path Block Verification for Faster Decoding in Speculative Sampling](https://mlsanity.com/paper/2602.16961/) (2026)
- [Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding](https://mlsanity.com/paper/2602.16994/) (2026)
- [CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration](https://mlsanity.com/paper/2605.11186/) (2026)
- [Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing](https://mlsanity.com/paper/2605.14978/) (2026)
- [Tsubame: Tree Replay for Diffusion-Based Speculative Decoding](https://mlsanity.com/paper/2609.33652/) (2026)
- [BV Loss: Block Verification-Aware Loss for Block Diffusion Speculative Decoding](https://mlsanity.com/paper/2609.34832/) (2026)
- [CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters](https://mlsanity.com/paper/2610.00321/) (2026)

## Abstract (arXiv)

Speculative decoding is an effective method for lossless acceleration of large language models during inference. It uses a fast model to draft a block of tokens which are then verified in parallel by the target model, and provides a guarantee that the output is distributed identically to a sample from the target model. In prior works, draft verification is performed independently token-by-token. Surprisingly, we show that this approach is not optimal. We propose Block Verification, a simple draft verification algorithm that verifies the entire block jointly and provides additional wall-clock speedup. We prove that the proposed mechanism is optimal in the expected number of tokens produced each iteration and specifically is never worse than the standard token-level verification. Empirically, block verification provides modest but consistent wall-clock speedups over the standard token verification algorithm of 5%-8% in a range of tasks and datasets. Given that block verification does not increase code complexity, maintains the strong lossless guarantee of the standard speculative decoding verification algorithm, cannot deteriorate performance, and, in fact, consistently improves it, it can be used as a good default in speculative decoding implementations.