mlsanityData to 7 Oct 2026

Trees from Marginals: Autoregressive drafting with factorized priors

arXiv 2607.06763 ↗First posted 7 Jul 2026cs.LG cs.CLcited by 2

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding greatly increases the interactivity of autoregressive language models by trading off computation for extra tokens generated in a single forward pass. Factorized draft models are especially efficient because they predict future-token marginals in parallel, but their independence assumption causes acceptance rates to degrade sharply as the speculative budget grows. We analyze this limitation and introduce Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top-K marginals of a factorized drafter. Weaver restores conditional dependencies between proposed tokens while avoiding a full-vocabulary projection. To support fast verification for models with Gated Delta Net layers, we derive a rollback-free tree-verification algorithm and implement optimized CUDA kernels in SGLang. By combining these model and systems contributions we achieve a 4.37-fold speedup over autoregressive decoding, and outperform a highly optimized DFlash baseline by 24.7%.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 53 of its 66 references that are in the map. First posted on arXiv 7 Jul 2026.

Builds on

53 of 66 references.

Closest ancestors · 25

Same direction, newest first

  1. 2026PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding9 May
  2. 2026Accelerating Speculative Decoding with Block Diffusion Draft Trees14 Apr
  3. 2026LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding27 Feb
  4. 2026Speculative Decoding with a Speculative Vocabulary14 Feb
  5. 2026DFlash: Block Diffusion for Flash Speculative Decoding5 Feb
  6. 2025VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMs28 Jun
  7. 2025STree: Speculative Tree Decoding for Hybrid State-Space Models20 May
  8. 2025Traversal Verification for Speculative Tree Decoding18 May
  9. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation23 Apr
  10. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling20 Feb
  11. 2024PLD+: Accelerating LLM inference by leveraging Language Model Artifacts2 Dec
  12. 2024SAM Decoding: Speculative Decoding via Suffix Automaton16 Nov
13 more
  1. 2024Multi-Draft Speculative Sampling: Canonical Decomposition and Theoretical Limits23 Oct
  2. 2024DySpec: Faster Speculative Decoding with Dynamic Token Tree Structure15 Oct
  3. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure25 Jun
  4. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  5. 2024Block Verification Accelerates Speculative Decoding15 Mar
  6. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  7. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  10. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  11. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  12. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  13. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  10. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  11. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  12. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
16 more
  1. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  2. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  3. 2025Muon is Scalable for LLM Trainingcited by 381
  4. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  5. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  6. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  7. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  8. 2014A* Samplingcited by 150
  9. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 128
  10. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  11. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 116
  12. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  13. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 48
  14. 2025ShareChat: A Dataset of Chatbot Conversations in the Wildcited by 12
  15. 2024Coupling without Communication and Drafter-Invariant Speculative Decodingcited by 6
  16. 2026Introspective Diffusion Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decodingcited by 2
  2. 2026LumoTree: Path-Parallel Speculative Verification for Hybrid Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TreeSpark: Calibrated, Load-Adaptive Draft Trees for Semi-Autoregressive Speculative Decoding
  2. 2026JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting
  3. 2026DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees
  4. 2026TAPS: Task Aware Proposal Distributions for Speculative Sampling
  5. 2026Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
  6. 2026PRESTO: Prefix-Aligned Tree Drafting for Diffusion Speculative Decoding
  7. 2026From Chains to Trees: Parent-Conditioned Drafting for Semi-Autoregressive Speculative Decoding
  8. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation