mlsanityData to 7 Oct 2026

Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding

arXiv 2402.05109 ↗First posted 7 Feb 2024cs.LGcited by 113

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

To combat the memory bandwidth-bound nature of autoregressive LLM inference, previous research has proposed the speculative decoding frame-work. To perform speculative decoding, a small draft model proposes candidate continuations of the input sequence that are then verified in parallel by the base model. One way to specify the draft model, as used in the recent Medusa decoding framework, is as a collection of lightweight heads, called draft heads, that operate on the base model's hidden states. To date, all existing draft heads have been sequentially independent, meaning that they speculate tokens in the candidate continuation independently of any preceding tokens in the candidate continuation. In this work, we propose Hydra heads: a sequentially-dependent drop-in replacement for standard draft heads that significantly improves the accuracy of draft head speculation. We further explore the design space of Hydra head training objectives and architectures, and propose a carefully tuned Hydra head recipe, which we call Hydra++, that improves decoding throughput by up to 1.31x and 2.70x compared to Medusa decoding and autoregressive de-coding respectively. Overall, Hydra heads are a simple and well-motivated intervention on standard draft heads that significantly improve the end-to-end speed of draft head-based speculative decoding. We make our code publicly available at https://github.com/zankner/Hydra.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 33 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

20 of 33 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  4. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  5. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  6. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  7. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 13

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  8. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  11. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  12. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
1 more
  1. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 47; 2026: 51.

113 papers.

Sort:

By area:

  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  2. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  3. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  4. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  5. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  6. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  7. 2024LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decodingcited by 46
  8. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  9. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 40
  10. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Modelscited by 37
  11. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 34
  12. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
38 more
  1. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  2. 2025Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decodingcited by 26
  3. 2025SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decodingcited by 26
  4. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  5. 2024Mixture of Attentions For Speculative Decodingcited by 17
  6. 2026D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Draftingcited by 16
  7. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decodingcited by 15
  8. 2024Speculative Decoding with CTC-based Draft Model for LLM Inference Accelerationcited by 14
  9. 2025Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decodingcited by 14
  10. 2025L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Modelscited by 14
  11. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 14
  12. 2025AutoJudge: Judge Decoding Without Manual Annotationcited by 13
  13. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  14. 2025Scaling Speculative Decoding with Lookahead Reasoningcited by 13
  15. 2025LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculationcited by 13
  16. 2025A Survey on Parallel Reasoningcited by 11
  17. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Matchcited by 11
  18. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarioscited by 11
  19. 2024Boosting Lossless Speculative Decoding via Feature Sampling and Partial Alignment Distillationcited by 10
  20. 2025Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decodingcited by 9
  21. 2025Faster and Better LLMs via Latency-Aware Test-Time Scalingcited by 9
  22. 2025Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decodingcited by 9
  23. 2025Fast and Expressive Multi-Byte Prediction with Probabilistic Circuitscited by 9
  24. 2025Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decodingcited by 9
  25. 2025DEER: Draft with Diffusion, Verify with Autoregressive Modelscited by 9
  26. 2024Accelerating Production LLMs with Combined Token/Embedding Speculatorscited by 8
  27. 2025Locality-aware Parallel Decoding for Efficient Autoregressive Image Generationcited by 8
  28. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decodingcited by 8
  29. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decodingcited by 7
  30. 2025AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelismcited by 7
  31. 2025R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoningcited by 7
  32. 2026PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decodingcited by 7
  33. 2024S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUscited by 6
  34. 2024KOALA: Enhancing Speculative Decoding for LLM via Multi-Layer Draft Heads with Adversarial Learningcited by 6
  35. 2025Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoEcited by 6
  36. 2025Understanding and Enhancing the Planning Capability of Language Models via Multi-Token Predictioncited by 6
  37. 2025Parallel Token Prediction for Language Modelscited by 6
  38. 2026Speculative Decoding with a Speculative Vocabularycited by 6

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Tutorial Proposal: Speculative Decoding for Efficient LLM Inference
  2. 2026Faster LLM Inference via Sequential Monte Carlo
  3. 2026From Chains to Trees: Parent-Conditioned Drafting for Semi-Autoregressive Speculative Decoding
  4. 2026DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inference
  5. 2026DLoop: Looped Speculative Decoding
  6. 2024Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference
  7. 2026AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
  8. 2025ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts