mlsanityData to 7 Oct 2026

OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure

arXiv 2406.17276 ↗First posted 25 Jun 2024cs.CLcited by 64

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Autoregressive language models demonstrate excellent performance in various scenarios. However, the inference efficiency is limited by its one-step-one-word generation mode, which has become a pressing problem recently as the models become increasingly larger. Speculative decoding employs a "draft and then verify" mechanism to allow multiple tokens to be generated in one step, realizing lossless acceleration. Existing methods mainly adopt fixed heuristic draft structures, which fail to adapt to different situations to maximize the acceptance length during verification. To alleviate this dilemma, we proposed OPT-Tree, an algorithm to construct adaptive and scalable draft trees. It searches the optimal tree structure that maximizes the mathematical expectation of the acceptance length in each decoding step. Experimental results reveal that OPT-Tree outperforms the existing draft structures and achieves a speed-up ratio of up to 3.2 compared with autoregressive decoding. If the draft model is powerful enough and the node budget is sufficient, it can generate more than ten tokens in a single step. Our code is available at https://github.com/Jikai0Wang/OPT-Tree.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 25 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

23 of 25 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement21 Feb
  2. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  3. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  4. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  6. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  7. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  8. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  9. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  10. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  11. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  12. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 11

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2024Mixtral of Expertscited by 1,785
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  9. 2022Efficiently Scaling Transformer Inferencecited by 529
  10. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  11. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 24; 2026: 35.

64 papers.

Sort:

By area:

  1. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  2. 2026Accelerating Speculative Decoding with Block Diffusion Draft Treescited by 27
  3. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  4. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 21
  5. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 20
  6. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Samplingcited by 20
  7. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  8. 2024AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structurescited by 17
  9. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  10. 2026D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Draftingcited by 16
  11. 2025TETRIS: Optimal Draft Token Selection for Batch Speculative Decodingcited by 14
  12. 2025LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculationcited by 13
38 more
  1. 2026TALON: Confidence-Aware Speculative Decoding with Adaptive Token Treescited by 12
  2. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarioscited by 11
  3. 2025Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verificationcited by 10
  4. 2025CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Draftercited by 9
  5. 2025Traversal Verification for Speculative Tree Decodingcited by 9
  6. 2025C2T: A Classifier-Based Tree Construction Method in Speculative Decodingcited by 8
  7. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decodingcited by 8
  8. 2024Faster Language Models with Better Multi-Token Prediction Using Tensor Decompositioncited by 7
  9. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decodingcited by 7
  10. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Servingcited by 6
  11. 2025SpecMamba: Accelerating Mamba Inference on FPGA with Speculative Decodingcited by 5
  12. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Modelscited by 4
  13. 2025SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequencescited by 4
  14. 2025Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMscited by 4
  15. 2025Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decodingcited by 4
  16. 2026TreeFlash: Parallel AR-Approximation for Faster Speculative Decodingcited by 4
  17. 2026D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decodingcited by 4
  18. 2026AngelSpec: Towards Real-World High Performance Inference with Speculative Decodingcited by 4
  19. 2025EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilizationcited by 3
  20. 2026Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMscited by 3
  21. 2026Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decodingcited by 3
  22. 2026SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Draftingcited by 3
  23. 2026Cost-Aware Diffusion Draft Trees for Speculative Decodingcited by 3
  24. 2025POSS: Position Specialist Generates Better Draft for Speculative Decodingcited by 2
  25. 2025Accelerate Speculative Decoding with Sparse Computation in Verificationcited by 2
  26. 2026PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Lengthcited by 2
  27. 2026Trees from Marginals: Autoregressive drafting with factorized priorscited by 2
  28. 2026PRESTO: Prefix-Aligned Tree Drafting for Diffusion Speculative Decodingcited by 2
  29. 2026From Chains to Trees: Parent-Conditioned Drafting for Semi-Autoregressive Speculative Decodingcited by 2
  30. 2025FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inferencecited by 1
  31. 2026MineDraft: A Framework for Batch Parallel Speculative Decodingcited by 1
  32. 2026Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Draftingcited by 1
  33. 2026SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decodingcited by 1
  34. 2026WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routingcited by 1
  35. 2026CURE: Local Uncertainty Repair for Block-Parallel Speculative Decodingcited by 1
  36. 2026Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decodingcited by 1
  37. 2026RheoSampling: Resolving the One-Hot Dilemma in Stochastic Dynamic-Tree Speculative Decodingcited by 1
  38. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inferencecited by 0

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting
  2. 2025Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding
  3. 2026UniVer: A Unified Perspective for Multi-step and Multi-draft Speculative Decoding
  4. 2026SMART: When is it Actually Worth Expanding a Speculative Tree?
  5. 2026When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding
  6. 2026GRAFT: Adaptive DLM-Based Draft Tree Construction with Target-Distilled Edge Scoring
  7. 2026Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding
  8. 2026DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees