mlsanityData to 14 Oct 2026

Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding

arXiv 2402.13720 ↗First posted 21 Feb 2024cs.CLcited by 25

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding is a widely used method that accelerates the generation process of large language models (LLMs) with no compromise in model performance. It achieves this goal by using an existing smaller model for drafting and then employing the target LLM to verify the draft in a low-cost parallel manner. Under such a drafting-verification framework, drafting efficiency has become a bottleneck in the final speedup of speculative decoding. Therefore, generating longer drafts at less cost can lead to better decoding speedup. To achieve this, we introduce Ouroboros, which can generate draft phrases to parallelize the drafting process and meanwhile lengthen drafts in a training-free manner. The experimental results on various typical text generation tasks show that Ouroboros can achieve speedups of up to 2.8× over speculative decoding and 3.9× over vanilla decoding, without fine-tuning draft and target models. The source code of Ouroboros is available at https://github.com/thunlp/Ouroboros.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 61 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

43 of 61 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  4. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  5. 2023PaSS: Parallel Speculative Sampling22 Nov
  6. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  7. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding9 Oct
  8. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  9. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  10. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  11. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  12. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 31

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
19 more
  1. 2015Teaching Machines to Read and Comprehendcited by 1,049
  2. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  3. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  4. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  5. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  6. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  7. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  8. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  9. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  10. 2019Depth-Adaptive Transformercited by 236
  11. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  12. 2019Structured Pruning of Large Language Modelscited by 203
  13. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  14. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  15. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
  16. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  17. 2019Imitation Learning for Non-Autoregressive Neural Machine Translationcited by 36
  18. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  19. 2020Know What You Don't Need: Single-Shot Meta-Pruning for Attention Headscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 9.

25 papers.

Sort:

By area:

  1. 2024Interpretable Contrastive Monte Carlo Tree Search Reasoningcited by 64
  2. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 51
  3. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  4. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 22
  5. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 21
  6. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Samplingcited by 20
  7. 2025SHANKS: Simultaneous Hearing and Thinking for Spoken Language Modelscited by 19
  8. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decodingcited by 14
  9. 2025Utility-Driven Speculative Decoding for Mixture-of-Expertscited by 13
  10. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Predictioncited by 13
  11. 2025BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsitycited by 6
  12. 2026Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decodingcited by 6
13 more
  1. 2024Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Trainingcited by 5
  2. 2025RLHFSpec: Breaking the Efficiency Bottleneck in RLHF Training via Adaptive Draftingcited by 5
  3. 2025Reject Only Critical Tokens: Pivot-Aware Speculative Decodingcited by 4
  4. 2024SSSD: Simply-Scalable Speculative Decodingcited by 2
  5. 2025FastCoder: Accelerating Repository-level Code Generation via Efficient Retrieval and Verificationcited by 2
  6. 2026PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Lengthcited by 2
  7. 2026Variational Speculative Decoding: Rethinking Draft Training from Token Likelihood to Sequence Acceptancecited by 2
  8. 2026TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rolloutscited by 2
  9. 2026Approximate Speculative Decodingcited by 1
  10. 2026Carryover Drafting: Recycling Rejected States for Speculative Decodingcited by 1
  11. 2026Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decodingcited by 0
  12. 2026Position-Aware Drafting for Inference Acceleration in LLM-Based Generative List-Wise Recommendationcited by 0
  13. 2026SP2ec: Adaptive Self-Speculative Decoding for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Decoding Speculative Decoding
  2. 2024FastDraft: How to Train Your Draft
  3. 2025Mamba Drafters for Speculative Decoding
  4. 2024Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration
  5. 2024SpecHub: Provable Acceleration to Multi-Draft Speculative Decoding
  6. 2025Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding
  7. 2026Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding
  8. 2025Speeding up Speculative Decoding via Sequential Approximate Verification