mlsanityData to 7 Oct 2026

REST: Retrieval-Based Speculative Decoding

arXiv 2311.08252 ↗First posted 14 Nov 2023cs.CL cs.AI cs.IRcited by 148

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

We introduce Retrieval-Based Speculative Decoding (REST), a novel algorithm designed to speed up language model generation. The key insight driving the development of REST is the observation that the process of text generation often includes certain common phases and patterns. Unlike previous methods that rely on a draft language model for speculative decoding, REST harnesses the power of retrieval to generate draft tokens. This method draws from the reservoir of existing knowledge, retrieving and employing relevant tokens based on the current context. Its plug-and-play nature allows for seamless integration and acceleration of any language models, all without necessitating additional training. When benchmarked on 7B and 13B language models in a single-batch setting, REST achieves a significant speedup of 1.62X to 2.36X on code or text generation. The code of REST is available at https://github.com/FasterDecoding/REST.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 41 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

29 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  2. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  3. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
14 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  7. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  11. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  12. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  13. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  14. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 45; 2025: 54; 2026: 49.

148 papers.

Sort:

By area:

  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  3. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  5. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  6. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  8. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  9. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  10. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  11. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  12. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
38 more
  1. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 64
  2. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoningcited by 60
  3. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  4. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  5. 2024Fast On-device LLM Inference with NPUscited by 58
  6. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 45
  7. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 40
  8. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 38
  9. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 35
  10. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 34
  11. 2024Block Verification Accelerates Speculative Decodingcited by 30
  12. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
  13. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
  14. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  15. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attributioncited by 26
  16. 2025Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decodingcited by 26
  17. 2024DySpec: Faster Speculative Decoding with Dynamic Token Tree Structurecited by 25
  18. 2025Efficient Reasoning for LLMs through Speculative Chain-of-Thoughtcited by 23
  19. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  20. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decodingcited by 21
  21. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 20
  22. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
  23. 2025Towards Optimal Multi-draft Speculative Decodingcited by 19
  24. 2025Towards AI Search Paradigmcited by 19
  25. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Servingcited by 18
  26. 2025BanditSpec: Adaptive Speculative Decoding via Bandit Algorithmscited by 18
  27. 2024Mixture of Attentions For Speculative Decodingcited by 17
  28. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 17
  29. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inferencecited by 16
  30. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decodingcited by 15
  31. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decodingcited by 14
  32. 2024BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Modelscited by 13
  33. 2024AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Accelerationcited by 13
  34. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculationcited by 13
  35. 2025AutoJudge: Judge Decoding Without Manual Annotationcited by 13
  36. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  37. 2025LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculationcited by 13
  38. 2024Grounding Language Models for Visual Entity Recognitioncited by 12

50 most cited of 148 All 148 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DReSD: Dense Retrieval for Speculative Decoding
  2. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  3. 2025Speculative Decoding: Performance or Illusion?
  4. 2025Fast Inference via Hierarchical Speculative Decoding
  5. 2024Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models
  6. 2026RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding
  7. 2024Decoding Speculative Decoding
  8. 2025Cacheback: Speculative Decoding With Nothing But Cache