mlsanityData to 7 Oct 2026

EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

arXiv 2606.27550 ↗First posted 25 Jun 2026cs.CL cs.LGcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Multi-token prediction has been shown to increase data density during training, improve downstream text-generation quality, and serves as the defacto approach for self-speculative decoding. Existing foundation and open source models that use MTP heads commit to a static tree-based attention topology throughout the entire generation sequence, meaning the speculation depth, and thus the compute required during verification, stays constant regardless of the context. This is fundamentally misaligned with the entropy patterns of natural language where low-entropy regions often support reliable multi-step drafting, while high-entropy regions require more conservative speculation. To address this, we propose Entropy-guided Multi-Token Prediction (EntMTP), a training-free scheduler that toggles between tree-based attention topologies from a set of task-specific pareto-optimal trees conditioned on a running estimate of local generation entropy. By matching speculation depth to context predictability, EntMTP maximizes expected accepted-token throughput across the full distribution of generated text without sacrificing generation quality. When evaluated across Humaneval, ShareGPT, GSM8k, and Litbench benchmarks, EntMTP consistently achieves a 1.15x speedup against Hydra and peak speedup of 1.36x against Medusa baselines respectively.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 6 of its 8 references that are in the map. First posted on arXiv 25 Jun 2026.

Builds on

6 of 8 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  2. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  4. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  5. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 1

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction
  2. 2025Entropy-Aware Token Rejection for Improving Speculative Decoding
  3. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
  4. 2026Think Twice Before You Write -- an Entropy-based Decoding Strategy to Enhance LLM Reasoning
  5. 2026Entropic-Time Inference: Self-Organizing Large Language Model Decoding Beyond Attention
  6. 2024Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference
  7. 2026AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction
  8. 2025On multi-token prediction for efficient LLM inference