Data to 7 Oct 2026

Blockwise Parallel Decoding for Deep Autoregressive Models

arXiv 1811.03115 ↗First posted 7 Nov 2018cs.LG cs.CL stat.MLcited by 318

Not on the map. Why?

Cited by 194 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep autoregressive sequence-to-sequence models have demonstrated impressive performance across a wide variety of tasks in recent years. While common architecture classes such as recurrent, convolutional, and self-attention networks make different trade-offs between the amount of computation needed per layer and the length of the critical path at training time, generation still remains an inherently sequential process. To overcome this limitation, we propose a novel blockwise parallel decoding scheme in which we make predictions for multiple time steps in parallel then back off to the longest prefix validated by a scoring model. This allows for substantial theoretical improvements in generation speed when applied to architectures that can process output sequences in parallel. We verify our approach empirically through a series of experiments using state-of-the-art self-attention models for machine translation and image super-resolution, achieving iteration reductions of up to 2x over a baseline greedy decoder with no loss in quality, or up to 7x in exchange for a slight decrease in performance. In terms of wall-clock time, our fastest models exhibit real-time speedups of up to 4x over standard greedy decoding.

Key facts. Cited by 318 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 6; 2023: 21; 2024: 96; 2025: 93; 2026: 99.

318 papers.

Sort:

By area:

2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
2024A Survey on Efficient Inference for Large Language Modelscited by 197

38 more

2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 160
2024The pitfalls of next-token predictioncited by 156
2023REST: Retrieval-Based Speculative Decodingcited by 148
2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 113
2023Parallel Sampling of Diffusion Modelscited by 112
2023Online Speculative Decodingcited by 109
2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
2025Universal Model Routing for Efficient LLM Inferencecited by 80
2022An Introduction to Neural Data Compressioncited by 78
2023Fairness in Serving Large Language Modelscited by 77
2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decodingcited by 74
2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 64
2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoningcited by 60
2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
2024Parallelized Autoregressive Visual Generationcited by 58
2022Future Transformer for Long-term Action Anticipationcited by 57
2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decodingcited by 53
2023PaSS: Parallel Speculative Samplingcited by 49
2024Do language models plan ahead for future tokens?cited by 47
2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 45
2024Post-Training Sparse Attention with Double Sparsitycited by 45
2021FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognitioncited by 43
2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43

50 most cited of 318 All 318 (JSON)