mlsanityData to 7 Oct 2026

A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models

arXiv 2405.13019 ↗First posted 15 May 2024cs.CL cs.AIcited by 5

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Despite the crucial importance of accelerating text generation in large language models (LLMs) for efficiently producing content, the sequential nature of this process often leads to high inference latency, posing challenges for real-time applications. Various techniques have been proposed and developed to address these challenges and improve efficiency. This paper presents a comprehensive survey of accelerated generation techniques in autoregressive language models, aiming to understand the state-of-the-art methods and their applications. We categorize these techniques into several key areas: speculative decoding, early exiting mechanisms, and non-autoregressive methods. We discuss each category's underlying principles, advantages, limitations, and recent advancements. Through this survey, we aim to offer insights into the current landscape of techniques in LLMs and provide guidance for future research directions in this critical area of natural language processing.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 62 of its 89 references that are in the map. First posted on arXiv 15 May 2024.

Builds on

62 of 89 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  2. 2024Block Verification Accelerates Speculative Decoding15 Mar
  3. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding19 Feb
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  5. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  6. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models28 Oct
  7. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  8. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding9 Oct
  9. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  10. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference5 Jul
  11. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  12. 2023Speculative Decoding with Big Little Decoder15 Feb
3 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  2. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  3. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2015Variational Inference with Normalizing Flowscited by 2,161
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2018Hierarchical Neural Story Generationcited by 1,182
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
35 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  4. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  5. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  6. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  7. 2019The State of Sparsity in Deep Neural Networkscited by 471
  8. 2017Non-Autoregressive Neural Machine Translationcited by 457
  9. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  10. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  11. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  12. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  13. 2019Depth-Adaptive Transformercited by 227
  14. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  15. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  16. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  17. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 103
  18. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  19. 2019FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flowcited by 89
  20. 2021Sparse is Enough in Scaling Transformerscited by 86
  21. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  22. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 63
  23. 2019Fast Structured Decoding for Sequence Modelscited by 59
  24. 2018Semi-Autoregressive Neural Machine Translationcited by 48
  25. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  26. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  27. 2018Non-Autoregressive Neural Machine Translation with Enhanced Decoder Inputcited by 39
  28. 2019Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translationcited by 39
  29. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
  30. 2021Non-Autoregressive Text Generation with Pre-trained Language Modelscited by 36
  31. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
  32. 2019Syntactically Supervised Transformers for Faster Neural Machine Translationcited by 19
  33. 2016Noisy Parallel Approximate Decoding for Conditional Recurrent Language Modelcited by 18
  34. 2023ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inferencecited by 13
  35. 2023SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inferencecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  2. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generationcited by 19
  3. 2025AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelismcited by 7
  4. 2025Parallel Token Prediction for Language Modelscited by 6
  5. 2026Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniques
  2. 2024Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines
  3. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  4. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies
  5. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
  6. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inference
  7. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  8. 2024Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding