mlsanityData to 14 Oct 2026

DReSD: Dense Retrieval for Speculative Decoding

arXiv 2502.15572 ↗First posted 21 Feb 2025cs.CLcited by 6

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding (SD) accelerates Large Language Model (LLM) generation by using an efficient draft model to propose the next few tokens, which are verified by the LLM in a single forward call, reducing latency while preserving its outputs. We focus on retrieval-based SD where the draft model retrieves the next tokens from a non-parametric datastore. Sparse retrieval (REST), which operates on the surface form of strings, is currently the dominant paradigm due to its simplicity and scalability. However, its effectiveness is limited due to the usage of short contexts and exact string matching. Instead, we introduce Dense Retrieval for Speculative Decoding (DReSD), a novel framework that uses approximate nearest neighbour search with contextualised token embeddings to retrieve the most semantically relevant token sequences for SD. Extensive experiments show that DReSD achieves (on average) 87% higher acceptance rates, 65% longer accepted tokens and 19% faster generation speeds compared to sparse retrieval (REST).

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 39 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

27 of 39 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding20 Nov
  2. 2024SAM Decoding: Speculative Decoding via Suffix Automaton16 Nov
  3. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation6 Nov
  4. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attribution29 May
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  7. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  8. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  12. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
7 more
  1. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  2. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  3. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  4. 2014A Tutorial on Principal Component Analysiscited by 141
  5. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  6. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 47
  7. 2024Accelerating Retrieval-Augmented Language Model Serving with Speculationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026Speculative Decoding with a Speculative Vocabularycited by 6
  2. 2025Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Executioncited by 5
  3. 2026FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibrationcited by 2
  4. 2026SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decodingcited by 1
  5. 2026AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decodingcited by 1
  6. 2026Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023REST: Retrieval-Based Speculative Decoding
  2. 2024CREST: Effectively Compacting a Datastore For Retrieval-Based Speculative Decoding
  3. 2025LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculation
  4. 2025RASD: Retrieval-Augmented Speculative Decoding
  5. 2026RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding
  6. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE
  7. 2024SSSD: Simply-Scalable Speculative Decoding
  8. 2025Speculative Decoding: Performance or Illusion?