mlsanityData to 7 Oct 2026

SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scaling

arXiv 2610.05106 ↗First posted 4 Oct 2026cs.AIcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Long-form reasoning makes inference expensive, and speculative decoding mitigates this cost by verifying multiple draft tokens in parallel. Its speedup, however, can fade as context grows and draft acceptance declines. We focus on attention-mass dilution: as softmax normalizes over more visible Keys, the mass concentrated on the highest-scoring Keys can decrease. We introduce SharpDraft, a training-free method that counteracts this effect through cardinality-aware Query scaling, without the computational overhead of online adaptation. Under explicit assumptions, we derive an exact top-k mass correction and deploy a closed-form fixed-slope approximation. Across AIME-26, GPQA-Diamond, and LongGenBench Diary, SharpDraft achieves 2.59-3.19× geometric-mean end-to-end speedups over target-only autoregressive decoding when applied to DFlash, PARD, and EAGLE 3.1. With DFlash, it improves decoding speed and outperforms full-parameter and LoRA-based online adaptation in end-to-end speedup, while matching the unmodified drafter's reported peak allocated GPU memory.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 28 references that are in the map. First posted on arXiv 4 Oct 2026.

Builds on

25 of 28 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Attention Drift: What Autoregressive Speculative Decoding Models Learn11 May
  2. 2026Test-Time Speculation10 May
  3. 2026When Drafts Evolve: Speculative Decoding Meets Online Learning13 Mar
  4. 2026DFlash: Block Diffusion for Flash Speculative Decoding5 Feb
  5. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation23 Apr
  6. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  7. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification24 Feb
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  9. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  10. 2023Online Speculative Decoding11 Oct
  11. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  12. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 13

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Let's Verify Step by Stepcited by 3,598
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  5. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  6. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  7. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 115
  8. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  9. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 47
  10. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 42
  11. 2025Scalable-Softmax Is Superior for Attentioncited by 38
  12. 2026AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rolloutscited by 8
1 more
  1. 2026LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
  2. 2026LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
  3. 2026A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
  4. 2024SSSD: Simply-Scalable Speculative Decoding
  5. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  6. 2025SPIRe: Boosting LLM Inference Throughput with Speculative Decoding
  7. 2025SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences
  8. 2026RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding