mlsanityData to 7 Oct 2026

Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity

arXiv 2410.01028 ↗First posted 1 Oct 2024cs.CLcited by 8

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

We present a simple on the fly method for faster inference of large language models. Unlike other (self-)speculative decoding techniques, our method does not require fine-tuning or black-box optimization to generate a fixed draft model, relying instead on simple rules to generate varying draft models adapted to the input context. We show empirically that our light-weight algorithm is competitive with the current SOTA for self-speculative decoding, while being a truly plug-and-play method.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 16 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

12 of 16 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting29 Apr
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  3. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  4. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 8

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  5. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  7. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  8. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  2. 2025KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimizationcited by 10
  3. 2025DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decodingcited by 8
  4. 2026KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problemcited by 4
  5. 2026ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decodingcited by 3
  6. 2025Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decodingcited by 1
  7. 2025Polybasic Speculative Decoding Through a Theoretical Perspectivecited by 1
  8. 2026Experience-Driven Dynamic Exits for LLMs with Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  2. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  4. 2024AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures
  5. 2025Self Speculative Decoding for Diffusion Large Language Models
  6. 2023Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding
  7. 2024SSSD: Simply-Scalable Speculative Decoding
  8. 2025CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs