mlsanityData to 14 Oct 2026

Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding

arXiv 2512.23858 ↗First posted 29 Dec 2025cs.LG cs.PLcited by 4

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding improves LLM inference by generating and verifying multiple tokens in parallel, but existing systems suffer from suboptimal performance due to a mismatch between dynamic speculation and static runtime assumptions. We present Yggdrasil, a co-designed system that enables latency-optimal speculative decoding through context-aware tree drafting and compiler-friendly execution. Yggdrasil introduces an equal-growth tree structure for static graph compatibility, a latency-aware optimization objective for draft selection, and stage-based scheduling to reduce overhead. Yggdrasil supports unmodified LLMs and achieves up to 3.98× speedup over state-of-the-art baselines across multiple hardware setups.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 51 references that are in the map. First posted on arXiv 29 Dec 2025.

Builds on

32 of 51 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
  2. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure25 Jun
  3. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attribution29 May
  4. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting29 Apr
  5. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding10 Apr
  6. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  7. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  10. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  11. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  12. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
1 more
  1. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  7. 2015Teaching Machines to Read and Comprehendcited by 1,049
  8. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  9. 2023A Survey on Model Compression for Large Language Modelscited by 269
  10. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  11. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  12. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
7 more
  1. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 109
  2. 2017DyNet: The Dynamic Neural Network Toolkitcited by 21
  3. 2023Think Outside the Code: Brainstorming Boosts Large Language Models in Code Generationcited by 19
  4. 2020Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learningcited by 17
  5. 2025M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Typecited by 16
  6. 2025VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inferencecited by 13
  7. 2023DyCL: Dynamic Neural Network Compilation Via Program Rewriting and Graph Optimizationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Bole: Efficient Tree Speculation for Hybrid-Attention Language Modelscited by 3
  2. 2026Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Draftingcited by 2
  3. 2026AREAL-DTA: Dynamic Tree Attention for Efficient Reinforcement Learning of Large Language Modelscited by 1
  4. 2026Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  2. 2023Accelerating LLM Inference with Staged Speculative Decoding
  3. 2026Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices
  4. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration
  5. 2026CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
  6. 2026BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration
  7. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  8. 2025FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference