# DReSD: Dense Retrieval for Speculative Decoding

arXiv: https://arxiv.org/abs/2502.15572 · HTML: https://mlsanity.com/paper/2502.15572/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 21 Feb 2025.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [A Tutorial on Principal Component Analysis](https://mlsanity.com/paper/1404.1100/) (2014)
- [Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)](https://mlsanity.com/paper/1405.5869/) (2014)
- [Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling](https://mlsanity.com/paper/2408.08696/) (2024)
- [SAM Decoding: Speculative Decoding via Suffix Automaton](https://mlsanity.com/paper/2411.10666/) (2024)
- [Nearest Neighbor Speculative Decoding for LLM Generation and Attribution](https://mlsanity.com/paper/2405.19325/) (2024)
- [Accelerating Retrieval-Augmented Language Model Serving with Speculation](https://mlsanity.com/paper/2401.14021/) (2024)
- [The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation](https://mlsanity.com/paper/2411.03786/) (2024)
- [Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding](https://mlsanity.com/paper/2411.13157/) (2024)

## Built on by (most cited first, 6 of 6)

- [Speculative Decoding with a Speculative Vocabulary](https://mlsanity.com/paper/2602.13836/) (2026)
- [Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Execution](https://mlsanity.com/paper/2510.15312/) (2025)
- [FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration](https://mlsanity.com/paper/2605.20022/) (2026)
- [SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding](https://mlsanity.com/paper/2606.00021/) (2026)
- [AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decoding](https://mlsanity.com/paper/2606.05742/) (2026)
- [Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes](https://mlsanity.com/paper/2608.03839/) (2026)

## Abstract (arXiv)

Speculative decoding (SD) accelerates Large Language Model (LLM) generation by using an efficient draft model to propose the next few tokens, which are verified by the LLM in a single forward call, reducing latency while preserving its outputs. We focus on retrieval-based SD where the draft model retrieves the next tokens from a non-parametric datastore. Sparse retrieval (REST), which operates on the surface form of strings, is currently the dominant paradigm due to its simplicity and scalability. However, its effectiveness is limited due to the usage of short contexts and exact string matching. Instead, we introduce Dense Retrieval for Speculative Decoding (DReSD), a novel framework that uses approximate nearest neighbour search with contextualised token embeddings to retrieve the most semantically relevant token sequences for SD. Extensive experiments show that DReSD achieves (on average) 87% higher acceptance rates, 65% longer accepted tokens and 19% faster generation speeds compared to sparse retrieval (REST).