mlsanityData to 14 Oct 2026

FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction

arXiv 2509.18362 ↗First posted 16 Sep 2025cs.LG cs.AIcited by 13

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

As large language models (LLMs) become increasingly powerful, the sequential nature of autoregressive generation creates a fundamental throughput bottleneck that limits the practical deployment. While Multi-Token Prediction (MTP) has demonstrated remarkable benefits for model training efficiency and performance, its inherent potential for inference acceleration remains largely unexplored. This paper introduces FastMTP, a simple yet effective method that improves multi-step draft quality by aligning MTP training with its inference pattern, significantly enhancing speculative decoding performance. Our approach fine-tunes a single MTP head with position-shared weights on self-distilled data, enabling it to capture dependencies among consecutive future tokens and maintain high acceptance rates across multiple recursive draft steps. By integrating language-aware dynamic vocabulary compression into the MTP head, we further reduce computational overhead in the drafting process. Experimental results across seven diverse benchmarks demonstrate that FastMTP achieves an average of 2.03x speedup compared to standard next token prediction with lossless output quality, outperforming vanilla MTP by 82%. FastMTP requires only lightweight training and seamlessly integrates with existing inference frameworks, offering a practical and rapidly deployable solution for accelerating LLM inference.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 64 of its 72 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

64 of 72 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  2. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling20 Feb
  3. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  4. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting29 Apr
  5. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  6. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding21 Feb
  7. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  10. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  11. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  12. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
4 more
  1. 2023Speculative Decoding with Big Little Decoder15 Feb
  2. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  3. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  4. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 48

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  10. 2024Qwen2 Technical Reportcited by 2,120
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  12. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
36 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  8. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  9. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  10. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  11. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  13. 2024A Survey on Large Language Models for Code Generationcited by 658
  14. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  15. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  16. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  17. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  18. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  19. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 384
  20. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  21. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  22. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  23. 2022Large Language Models Are Reasoning Teacherscited by 329
  24. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  25. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
  26. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 279
  27. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  28. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  29. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  30. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 178
  31. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuningcited by 169
  32. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 121
  33. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  34. 2025Efficient Reasoning Models: A Surveycited by 95
  35. 2025LongCat-Flash Technical Reportcited by 58
  36. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2026LK Losses: Direct Acceptance Rate Optimization for Speculative Decodingcited by 18
  2. 2026Multi-Token Prediction via Self-Distillationcited by 7
  3. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  4. 2025xLLM Technical Reportcited by 3
  5. 2026Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputscited by 3
  6. 2026Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Trainingcited by 3
  7. 2026Efficient Document Parsing via Parallel Token Predictioncited by 2
  8. 2026Temporal Guidance for Large Language Modelscited by 0
  9. 2026Latent Action Reparameterization for Efficient Agent Inferencecited by 0
  10. 2026EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptationcited by 0
  11. 2026Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detectioncited by 0
  12. 2026Jina-OCR-v1: Efficient Document Parsing with Speculative Decoding and Dense Verifiable Rewardscited by 0
1 more
  1. 2026SEED: Self-Speculative Decoding via Implicit Encoder-Decodercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On multi-token prediction for efficient LLM inference
  2. 2025QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
  3. 2024FIRP: Faster LLM inference via future intermediate representation prediction
  4. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  5. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation
  6. 2026Self-Distillation for Multi-Token Prediction
  7. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  8. 2024Accelerating Production LLMs with Combined Token/Embedding Speculators