mlsanityData to 14 Oct 2026

Recurrent Drafter for Fast Speculative Decoding in Large Language Models

arXiv 2403.09919 ↗First posted 14 Mar 2024cs.CL cs.LGcited by 37

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

We present Recurrent Drafter (ReDrafter), an advanced speculative decoding approach that achieves state-of-the-art speedup for large language models (LLMs) inference. The performance gains are driven by three key aspects: (1) leveraging a recurrent neural network (RNN) as the draft model conditioning on LLM's hidden states, (2) applying a dynamic tree attention algorithm over beam search results to eliminate duplicated prefixes in candidate sequences, and (3) training through knowledge distillation from the LLM. ReDrafter accelerates Vicuna inference in MT-Bench by up to 2.8x with a PyTorch implementation on Nvidia H100 GPUs. To demonstrate its practicality in real environments, we also validated its effectiveness for on-device applications by implementing the approach in MLX and benchmarking performance on Metal GPUs in Apple Silicon chips, achieving up to 2.3x speedup.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 42 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

29 of 42 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  3. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  5. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  6. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  7. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  8. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  9. 2023Online Speculative Decoding11 Oct
  10. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  11. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  12. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
2 more
  1. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  2. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 15

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2023PaLM 2 Technical Reportcited by 1,162
  9. 2016Sequence-Level Knowledge Distillationcited by 801
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  11. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  12. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
3 more
  1. 2022Truncation Sampling as Language Model Desmoothingcited by 123
  2. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 65
  3. 2021Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Trainingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 9; 2026: 16.

37 papers.

Sort:

By area:

  1. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  3. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 75
  4. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  5. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  6. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  7. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 47
  8. 2024DySpec: Faster Speculative Decoding with Dynamic Token Tree Structurecited by 25
  9. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  10. 2026D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Draftingcited by 17
  11. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decodingcited by 16
  12. 2024Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledgecited by 10
25 more
  1. 2025Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decodingcited by 10
  2. 2024Adaptive Draft-Verification for Efficient Large Language Model Decodingcited by 9
  3. 2024Accelerating Production LLMs with Combined Token/Embedding Speculatorscited by 8
  4. 2024FastDraft: How to Train Your Draftcited by 6
  5. 2024Clover-2: Accurate Inference for Regressive Lightweight Speculative Decodingcited by 5
  6. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 5
  7. 2025TokenSwift: Lossless Acceleration of Ultra Long Sequence Generationcited by 4
  8. 2025WhisperKit: On-device Real-time ASR with Billion-Scale Transformerscited by 4
  9. 2026DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Draftingcited by 4
  10. 2026SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Draftingcited by 3
  11. 2024SSSD: Simply-Scalable Speculative Decodingcited by 2
  12. 2025Hierarchical Verification of Speculative Beams for Accelerating LLM Inferencecited by 1
  13. 2026TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inferencecited by 1
  14. 2026MineDraft: A Framework for Batch Parallel Speculative Decodingcited by 1
  15. 2026H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cachecited by 1
  16. 2025Cross-Attention Speculative Decodingcited by 0
  17. 2026Cross-Family Speculative Decoding for Polish Language Models on Apple~Silicon: An Empirical Evaluation of Bielik~11B with UAG-Extended MLX-LMcited by 0
  18. 2026EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptationcited by 0
  19. 2026CONCORD: Asynchronous Sparse Aggregation for Device-Cloud RAG under Document Isolationcited by 0
  20. 2026Token-Operations-Oriented Inference Optimization Techniques for Large Modelscited by 0
  21. 2026Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detectioncited by 0
  22. 2026LongSpark: Efficient speculative decoding with a fixed-cost parallel draftercited by 0
  23. 2026JevSpawn: Adaptive Agentic Inference through Compositional Action Spacescited by 0
  24. 2026Where Draft Trees Lose Target Mass: Exit-Guided Speculative Decodingcited by 0
  25. 2026Contextual Correction of Parallel Draftsfor Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
  2. 2026Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
  3. 2026SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding
  4. 2026Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
  5. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inference
  6. 2026D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
  7. 2025SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decoding
  8. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models