# Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential

arXiv: https://arxiv.org/abs/2507.11851 · HTML: https://mlsanity.com/paper/2507.11851/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 31 references that are in the map.
- First posted on arXiv 16 Jul 2025.

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [Better & Faster Large Language Models via Multi-token Prediction](https://mlsanity.com/paper/2404.19737/) (2024)
- [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://mlsanity.com/paper/2503.01840/) (2025)
- [DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation](https://mlsanity.com/paper/2506.20639/) (2025)
- [DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://mlsanity.com/paper/2310.08461/) (2023)
- [Accelerating LLM Inference with Staged Speculative Decoding](https://mlsanity.com/paper/2308.04623/) (2023)
- [Understanding Knowledge Distillation in Non-autoregressive Machine Translation](https://mlsanity.com/paper/1911.02727/) (2019)
- [Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding](https://mlsanity.com/paper/2402.05109/) (2024)
- [PaSS: Parallel Speculative Sampling](https://mlsanity.com/paper/2311.13581/) (2023)
- [Speculative Streaming: Fast LLM Inference without Auxiliary Models](https://mlsanity.com/paper/2402.11131/) (2024)
- [Recurrent Drafter for Fast Speculative Decoding in Large Language Models](https://mlsanity.com/paper/2403.09919/) (2024)
- [ParallelSpec: Parallel Drafter for Efficient Speculative Decoding](https://mlsanity.com/paper/2410.05589/) (2024)
- [Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference](https://mlsanity.com/paper/2405.18628/) (2024)
- [Multi-Token Prediction Needs Registers](https://mlsanity.com/paper/2505.10518/) (2025)
- [MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2405.18897/) (2024)
- [SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens](https://mlsanity.com/paper/2403.18647/) (2024)

## Built on by (most cited first, 30 of 47)

- [DFlash: Block Diffusion for Flash Speculative Decoding](https://mlsanity.com/paper/2602.06036/) (2026)
- [A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models](https://mlsanity.com/paper/2508.08712/) (2025)
- [LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding](https://mlsanity.com/paper/2605.27365/) (2026)
- [Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding](https://mlsanity.com/paper/2607.05722/) (2026)
- [Sequential Diffusion Language Models](https://mlsanity.com/paper/2509.24007/) (2025)
- [Speculative Speculative Decoding](https://mlsanity.com/paper/2603.03251/) (2026)
- [Enabling Approximate Joint Sampling in Diffusion LMs](https://mlsanity.com/paper/2509.22738/) (2025)
- [LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation](https://mlsanity.com/paper/2603.10899/) (2026)
- [Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLM](https://mlsanity.com/paper/2604.06832/) (2026)
- [Introspective Diffusion Language Models](https://mlsanity.com/paper/2604.11035/) (2026)
- [Parallel Token Prediction for Language Models](https://mlsanity.com/paper/2512.21323/) (2025)
- [Multi-Token Prediction via Self-Distillation](https://mlsanity.com/paper/2602.06019/) (2026)
- [Understanding and Enhancing the Planning Capability of Language Models via Multi-Token Prediction](https://mlsanity.com/paper/2509.23186/) (2025)
- [NEZHA: A Zero-sacrifice and Hyperspeed Decoding Architecture for Generative Recommendations](https://mlsanity.com/paper/2511.18793/) (2025)
- [Latent Chain-of-Thought as Planning: Decoupling Reasoning from Verbalization](https://mlsanity.com/paper/2601.21358/) (2026)
- [Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance](https://mlsanity.com/paper/2602.01047/) (2026)
- [Steering Pretrained Drafters during Speculative Decoding](https://mlsanity.com/paper/2511.09844/) (2025)
- [Self-Distillation for Multi-Token Prediction](https://mlsanity.com/paper/2603.23911/) (2026)
- [NITP: Next Implicit Token Prediction for LLM Pre-training](https://mlsanity.com/paper/2605.24956/) (2026)
- [Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs](https://mlsanity.com/paper/2605.12460/) (2026)
- [PRESTO: Prefix-Aligned Tree Drafting for Diffusion Speculative Decoding](https://mlsanity.com/paper/2607.22634/) (2026)
- [Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference](https://mlsanity.com/paper/2510.13161/) (2025)
- [Global Resolution: Optimal Multi-Draft Speculative Sampling via Convex Minimization](https://mlsanity.com/paper/2511.15898/) (2025)
- [FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration](https://mlsanity.com/paper/2605.20022/) (2026)
- [Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains](https://mlsanity.com/paper/2605.25745/) (2026)
- [Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting](https://mlsanity.com/paper/2605.29727/) (2026)
- [BlockPilot: Instance-Adaptive Policy Learning for Diffusion-based Speculative Decoding](https://mlsanity.com/paper/2606.31315/) (2026)
- [Unlocking Lossless Speedups in LLMs via Discrete Diffusion](https://mlsanity.com/paper/2609.04010/) (2026)
- [One-Token Verification for Reasoning Correctness Estimation](https://mlsanity.com/paper/2603.01025/) (2026)
- [Thinking into the Future: Latent Lookahead Training for Transformers](https://mlsanity.com/paper/2603.20219/) (2026)

## Abstract (arXiv)

Autoregressive language models are constrained by their inherently sequential nature, generating one token at a time. This paradigm limits inference speed and parallelism, especially during later stages of generation when the direction and semantics of text are relatively certain. In this work, we propose a novel framework that leverages the inherent knowledge of vanilla autoregressive language models about future tokens, combining techniques to realize this potential and enable simultaneous prediction of multiple subsequent tokens. Our approach introduces several key innovations: (1) a masked-input formulation where multiple future tokens are jointly predicted from a common prefix; (2) a gated LoRA formulation that preserves the original LLM's functionality, while equipping it for multi-token prediction; (3) a lightweight, learnable sampler module that generates coherent sequences from the predicted future tokens; (4) a set of auxiliary training losses, including a consistency loss, to enhance the coherence and accuracy of jointly generated tokens; and (5) a speculative generation strategy that expands tokens quadratically in the future while maintaining high fidelity. Our method achieves significant speedups through supervised fine-tuning on pretrained models. For example, it generates code and math nearly 5x faster, and improves general chat and knowledge tasks by almost 2.5x. These gains come without any loss in quality.