mlsanityData to 7 Oct 2026

Faster LLM Inference via Sequential Monte Carlo

arXiv 2604.15672 ↗First posted 17 Apr 2026cs.LG cs.CLcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding (SD) accelerates language model inference by drafting tokens from a cheap proposal model and verifying them against an expensive target model via rejection sampling. Because rejection truncates the draft block at the first error, throughput degrades when draft and target diverge. Rather than rejecting draft tokens outright, we propose to reweight them. To this end, we introduce sequential Monte Carlo speculative decoding (SMC-SD), which replaces token-level rejection with importance-weighted resampling over a population of draft particles. SMC-SD is a principled approximate inference scheme that trades exactness for additional speed, while preserving theoretical bounds on its per-step approximation error. Because LLM inference is memory bandwidth-bound, the arithmetic needed to draft particles and to score them in parallel comes nearly for free -- SMC-SD uses idle compute to turn verification into a vectorized, fixed-size operation with no rollback. Empirically, SMC-SD achieves 2.36x speed-up over speculative decoding and a 5.2x speed-up over autoregressive decoding, while remaining within 3% of the target model's accuracy on reasoning, instruction-following, and coding benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 33 references that are in the map. First posted on arXiv 17 Apr 2026.

Builds on

23 of 33 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026DFlash: Block Diffusion for Flash Speculative Decoding5 Feb
  2. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match28 Nov
  3. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  4. 2025Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoff28 Feb
  5. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment31 Jan
  6. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  7. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths30 May
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  10. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  11. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 12

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  4. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  5. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  6. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 84
  7. 2023Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programscited by 79
  8. 2019Elements of Sequential Monte Carlocited by 60
  9. 2026FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scalingcited by 58
  10. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 45
  11. 2026Power-SMC: Low-Latency Sequence-Level Power Sampling for Training-Free LLM Reasoningcited by 17
  12. 2025Syntactic Control of Language Models by Posterior Inferencecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  2. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  3. 2025Tutorial Proposal: Speculative Decoding for Efficient LLM Inference
  4. 2025Speculative Decoding: Performance or Illusion?
  5. 2023Accelerating LLM Inference with Staged Speculative Decoding
  6. 2024A Theoretical Perspective for Speculative Decoding Algorithm
  7. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
  8. 2025Speeding up Speculative Decoding via Sequential Approximate Verification