mlsanityData to 14 Oct 2026

Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling

arXiv 2604.04987 ↗First posted 5 Apr 2026cs.LG cs.AI math.OCcited by 4

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative sampling (SpS) has been successful in accelerating the decoding throughput of auto-regressive large language models by leveraging smaller draft models. SpS strictly enforces the generated distribution to match that of the verifier LLM. This is unnecessarily restrictive as slight variations of the verifier's distribution, such as sampling with top-k or temperature, would also be acceptable. Typical acceptance sampling (TAS) alleviates this issue by accepting more tokens using entropy-based heuristics. However, this approach distorts the verifier distribution, potentially degrading output quality when the verifier encodes critical information. In this work, we formalize the speculative sampling algorithm through the lens of constrained optimization. Based on this formulation, we propose Cactus (constrained acceptance speculative sampling), a method that guarantees controlled divergence from the verifier distribution and increasing acceptance rates. Empirical results across a wide range of benchmarks confirm the effectiveness of our approach.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 52 of its 58 references that are in the map. First posted on arXiv 5 Apr 2026.

Builds on

52 of 58 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  2. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniques27 Feb
  3. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoning31 Jan
  4. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment31 Jan
  5. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  6. 2024Faster Cascades via Speculative Decoding29 May
  7. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting29 Apr
  8. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement21 Feb
  9. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  10. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  11. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  12. 2024Multi-Candidate Speculative Decoding12 Jan
5 more
  1. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  2. 2023Speculative Decoding with Big Little Decoder15 Feb
  3. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  4. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  5. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
23 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  3. 2020Rethinking Attention with Performerscited by 1,601
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  5. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  9. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  10. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  11. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  12. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 213
  13. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  14. 2021Self-attention Does Not Need O(n2) Memorycited by 154
  15. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  16. 2022Truncation Sampling as Language Model Desmoothingcited by 123
  17. 2020If beam search is the answer, what was the question?cited by 116
  18. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 114
  19. 2022Locally Typical Samplingcited by 111
  20. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  21. 2025Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Timecited by 48
  22. 2025Ultra-Low-Dimensional Prompt Tuning via Random Projectioncited by 2
  23. 2025Radar: Fast Long-Context Decoding for Any Transformercited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026A Practical Investigation of Training-free Relaxed Speculative Decodingcited by 3
  2. 2026Approximate Speculative Decodingcited by 1
  3. 2026Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Headscited by 0
  4. 2026Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distributioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding
  2. 2026Acceptance-Aware Draft Model Training for Speculative Decoding
  3. 2026D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
  4. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism
  5. 2024A Theoretical Perspective for Speculative Decoding Algorithm
  6. 2025Draft, Verify, and Improve: Toward Training-Aware Speculative Decoding
  7. 2023SpecTr: Fast Speculative Decoding via Optimal Transport
  8. 2026LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization