# On multi-token prediction for efficient LLM inference

arXiv: https://arxiv.org/abs/2502.09419 · HTML: https://mlsanity.com/paper/2502.09419/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- First posted on arXiv 13 Feb 2025.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Built on by (most cited first, 7 of 7)

- [A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models](https://mlsanity.com/paper/2508.08712/) (2025)
- [L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models](https://mlsanity.com/paper/2505.17505/) (2025)
- [KORMo: Korean Open Reasoning Model for Everyone](https://mlsanity.com/paper/2510.09426/) (2025)
- [Self-Distillation for Multi-Token Prediction](https://mlsanity.com/paper/2603.23911/) (2026)
- [Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing](https://mlsanity.com/paper/2603.17942/) (2026)
- [Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning](https://mlsanity.com/paper/2608.14290/) (2026)
- [SEED: Self-Speculative Decoding via Implicit Encoder-Decoder](https://mlsanity.com/paper/2609.36590/) (2026)

## Abstract (arXiv)

We systematically investigate multi-token prediction (MTP) capabilities within LLMs pre-trained for next-token prediction (NTP). We first show that such models inherently possess MTP capabilities via numerical marginalization over intermediate token probabilities, though performance is data-dependent and improves with model scale. Furthermore, we explore the challenges of integrating MTP heads into frozen LLMs and find that their hidden layers are strongly specialized for NTP, making adaptation non-trivial. Finally, we show that while joint training of MTP heads with the backbone improves performance, it cannot fully overcome this barrier, prompting further research in this direction. Our findings provide a deeper understanding of MTP applied to pretrained LLMs, informing strategies for accelerating inference through parallel token prediction.