mlsanityData to 14 Oct 2026

TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding

arXiv 2502.15197 ↗First posted 21 Feb 2025cs.CL cs.AIcited by 15

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

We propose TETRIS, a novel method that optimizes the total throughput of batch speculative decoding in multi-request settings. Unlike existing methods that optimize for a single request or a group of requests as a whole, TETRIS actively selects the most promising draft tokens (for every request in a batch) to be accepted when verified in parallel, resulting in fewer rejected tokens and hence less wasted computing resources. Such an effective resource utilization to achieve fast inference in large language models (LLMs) is especially important to service providers with limited inference capacity. Compared to baseline speculative decoding, TETRIS yields a consistently higher acceptance rate and more effective utilization of the limited inference capacity. We show theoretically and empirically that TETRIS outperforms baseline speculative decoding and existing methods that dynamically select draft tokens, leading to a more efficient batch inference in LLMs.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 36 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

22 of 36 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding20 Nov
  2. 2024AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probability24 Oct
  3. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  4. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure25 Jun
  5. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  6. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput20 Jun
  7. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths30 May
  8. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution24 Feb
  9. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  10. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  11. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  12. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
3 more
  1. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  2. 2023Speculative Decoding with Big Little Decoder15 Feb
  3. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 7

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2023Large Language Models for Robotics: A Surveycited by 133
  6. 2024Efficient LLM Scheduling by Learning to Rankcited by 85
  7. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarioscited by 12
  2. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Expertscited by 11
  3. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Servingcited by 6
  4. 2026D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decodingcited by 5
  5. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decodingcited by 3
  6. 2025Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problemcited by 3
  7. 2026MineDraft: A Framework for Batch Parallel Speculative Decodingcited by 1
  8. 2026FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Servingcited by 1
  9. 2026Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inferencecited by 1
  10. 2026The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Expertscited by 1
  11. 2025Uncovering Scaling Laws for Large Language Models via Inverse Problemscited by 0
  12. 2026AI-Model Network: Concept, Current State and Futurecited by 0
3 more
  1. 2026Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detectioncited by 0
  2. 2026CAST: Cost-Aware Speculative Trees from One-Pass Block Drafterscited by 0
  3. 2026SP2ec: Adaptive Self-Speculative Decoding for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Speculative Decoding: Performance or Illusion?
  2. 2025Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs
  3. 2024Decoding Speculative Decoding
  4. 2025TapOut: A Bandit-Based Approach to Dynamic Speculative Decoding
  5. 2026LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization
  6. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models
  7. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  8. 2026An Interpretable Latency Model for Speculative Decoding in LLM Serving