mlsanityData to 7 Oct 2026

Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

arXiv 2401.10774 ↗First posted 19 Jan 2024cs.LG cs.CLcited by 821

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large Language Models (LLMs) employ auto-regressive decoding that requires sequential computation, with each step reliant on the previous one's output. This creates a bottleneck as each step necessitates moving the full model parameters from High-Bandwidth Memory (HBM) to the accelerator's cache. While methods such as speculative decoding have been suggested to address this issue, their implementation is impeded by the challenges associated with acquiring and maintaining a separate draft model. In this paper, we present Medusa, an efficient method that augments LLM inference by adding extra decoding heads to predict multiple subsequent tokens in parallel. Using a tree-based attention mechanism, Medusa constructs multiple candidate continuations and verifies them simultaneously in each decoding step. By leveraging parallel processing, Medusa substantially reduces the number of decoding steps required. We present two levels of fine-tuning procedures for Medusa to meet the needs of different use cases: Medusa-1: Medusa is directly fine-tuned on top of a frozen backbone LLM, enabling lossless inference acceleration. Medusa-2: Medusa is fine-tuned together with the backbone LLM, enabling better prediction accuracy of Medusa heads and higher speedup but needing a special training recipe that preserves the backbone model's capabilities. Moreover, we propose several extensions that improve or expand the utility of Medusa, including a self-distillation to handle situations where no training data is available and a typical acceptance scheme to boost the acceptance rate while maintaining generation quality. We evaluate Medusa on models of various sizes and training procedures. Our experiments demonstrate that Medusa-1 can achieve over 2.2x speedup without compromising generation quality, while Medusa-2 further improves the speedup to 2.3-3.6x.

Key facts. Cited by 821 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 51 references that are in the map. First posted on arXiv 19 Jan 2024.

Builds on

36 of 51 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  2. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  3. 2023Online Speculative Decoding11 Oct
  4. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  5. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  6. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
18 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  3. 2018Hierarchical Neural Story Generationcited by 1,182
  4. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  7. 2016Sequence-Level Knowledge Distillationcited by 761
  8. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  10. 2022Efficiently Scaling Transformer Inferencecited by 529
  11. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  12. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  13. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  14. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  15. 2022Truncation Sampling as Language Model Desmoothingcited by 121
  16. 2022Locally Typical Samplingcited by 108
  17. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  18. 2022On the probability-quality paradox in language generationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 156; 2025: 293; 2026: 372.

821 papers.

Sort:

By area:

  1. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  2. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  3. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  4. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  5. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  6. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  7. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  8. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  9. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  10. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  11. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  12. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
38 more
  1. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  2. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  3. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  4. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  5. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  6. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 120
  7. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 113
  8. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  9. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  10. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  11. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  12. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  13. 2025MemGen: Weaving Generative Latent Memory for Self-Evolving Agentscited by 99
  14. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  15. 2024Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scalecited by 87
  16. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 86
  17. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Lengthcited by 83
  18. 2024RandAR: Decoder-only Autoregressive Visual Generation in Random Orderscited by 82
  19. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  20. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  21. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  22. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
  23. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  24. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  25. 2024CLLMs: Consistency Large Language Modelscited by 71
  26. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  27. 2024Learning Harmonized Representations for Speculative Samplingcited by 69
  28. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  29. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  30. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  31. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 64
  32. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  33. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoningcited by 60
  34. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  35. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  36. 2024Fast On-device LLM Inference with NPUscited by 58
  37. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  38. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52

50 most cited of 821 All 821 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
  2. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  3. 2023Efficient LLM Inference on CPUs
  4. 2024SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens
  5. 2024Progressive Mixed-Precision Decoding for Efficient LLM Inference
  6. 2023Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding
  7. 2026LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inference
  8. 2026FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference