mlsanityData to 14 Oct 2026

KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimization

arXiv 2505.16162 ↗First posted 22 May 2025cs.CLcited by 11

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative Decoding (SD) has emerged as a widely used paradigm to accelerate the inference of large language models (LLMs) without compromising generation quality. It works by efficiently drafting multiple tokens using a compact model and then verifying them in parallel using the target LLM. Notably, Self-Speculative Decoding proposes skipping certain layers to construct the draft model, which eliminates the need for additional parameters or training. Despite its strengths, we observe in this work that drafting with layer skipping exhibits significant sensitivity to domain shifts, leading to a substantial drop in acceleration performance. To enhance the domain generalizability of this paradigm, we introduce KNN-SSD, an algorithm that leverages K-Nearest Neighbor (KNN) search to match different skipped layers with various domain inputs. We evaluated our algorithm in various models and multiple tasks, observing that its application leads to 1.3x-1.6x speedup in LLM inference.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 38 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

29 of 38 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniques27 Feb
  2. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment31 Jan
  3. 2024Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration25 Nov
  4. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration9 Oct
  5. 2024Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity1 Oct
  6. 2024Learning Harmonized Representations for Speculative Sampling28 Aug
  7. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  8. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  10. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  11. 2023Speculative Decoding with Big Little Decoder15 Feb
  12. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
2 more
  1. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  2. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 15

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  6. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  7. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  8. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  9. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  10. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  11. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  12. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
3 more
  1. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  2. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 105
  3. 2024Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategycited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Matchcited by 10
  2. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  3. 2026Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decodingcited by 6
  4. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 6
  5. 2026When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?cited by 5
  6. 2026HIPPO: Accelerating Video Large Language Models Inference via Holistic-aware Parallel Speculative Decodingcited by 3
  7. 2026ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decodingcited by 3
  8. 2026TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rolloutscited by 2
  9. 2026Experience-Driven Dynamic Exits for LLMs with Reinforcement Learningcited by 0
  10. 2026TSS: Target-Side Sparsification for Speculative Decoding in Domain-Specific Large Language Modelscited by 0
  11. 2026S3R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism
  2. 2025CLaSp: In-Context Layer Skip for Self-Speculative Decoding
  3. 2026KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem
  4. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
  5. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  6. 2023Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding
  7. 2024SSSD: Simply-Scalable Speculative Decoding
  8. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling