mlsanityData to 14 Oct 2026

Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential

arXiv 2507.11851 ↗First posted 16 Jul 2025cs.CL cs.LGcited by 47

Not on the map. Why?

Cited by 20 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Autoregressive language models are constrained by their inherently sequential nature, generating one token at a time. This paradigm limits inference speed and parallelism, especially during later stages of generation when the direction and semantics of text are relatively certain. In this work, we propose a novel framework that leverages the inherent knowledge of vanilla autoregressive language models about future tokens, combining techniques to realize this potential and enable simultaneous prediction of multiple subsequent tokens. Our approach introduces several key innovations: (1) a masked-input formulation where multiple future tokens are jointly predicted from a common prefix; (2) a gated LoRA formulation that preserves the original LLM's functionality, while equipping it for multi-token prediction; (3) a lightweight, learnable sampler module that generates coherent sequences from the predicted future tokens; (4) a set of auxiliary training losses, including a consistency loss, to enhance the coherence and accuracy of jointly generated tokens; and (5) a speculative generation strategy that expands tokens quadratically in the future while maintaining high fidelity. Our method achieves significant speedups through supervised fine-tuning on pretrained models. For example, it generates code and math nearly 5x faster, and improves general chat and knowledge tasks by almost 2.5x. These gains come without any loss in quality.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 27 of its 31 references that are in the map. First posted on arXiv 16 Jul 2025.

Builds on

27 of 31 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  5. 2025Large Language Diffusion Modelscited by 912
  6. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 846
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  8. 2016Sequence-Level Knowledge Distillationcited by 801
  9. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  10. 2017Non-Autoregressive Neural Machine Translationcited by 463
  11. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  12. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
15 more
  1. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  2. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  3. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  4. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 165
  5. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  6. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 122
  7. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 117
  8. 2023PaSS: Parallel Speculative Samplingcited by 49
  9. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Modelscited by 38
  10. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 37
  11. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 22
  12. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inferencecited by 16
  13. 2025Multi-Token Prediction Needs Registerscited by 16
  14. 2024MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuningcited by 6
  15. 2024SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokenscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 38.

47 papers.

Sort:

By area:

  1. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 135
  2. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  3. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  4. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 24
  5. 2025Sequential Diffusion Language Modelscited by 19
  6. 2026Speculative Speculative Decodingcited by 15
  7. 2025Enabling Approximate Joint Sampling in Diffusion LMscited by 10
  8. 2026LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generationcited by 10
  9. 2026Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLMcited by 9
  10. 2026Introspective Diffusion Language Modelscited by 8
  11. 2025Parallel Token Prediction for Language Modelscited by 7
  12. 2026Multi-Token Prediction via Self-Distillationcited by 7
35 more
  1. 2025Understanding and Enhancing the Planning Capability of Language Models via Multi-Token Predictioncited by 6
  2. 2025NEZHA: A Zero-sacrifice and Hyperspeed Decoding Architecture for Generative Recommendationscited by 6
  3. 2026Latent Chain-of-Thought as Planning: Decoupling Reasoning from Verbalizationcited by 6
  4. 2026Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidancecited by 5
  5. 2025Steering Pretrained Drafters during Speculative Decodingcited by 4
  6. 2026Self-Distillation for Multi-Token Predictioncited by 4
  7. 2026NITP: Next Implicit Token Prediction for LLM Pre-trainingcited by 4
  8. 2026Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputscited by 3
  9. 2026PRESTO: Prefix-Aligned Tree Drafting for Diffusion Speculative Decodingcited by 3
  10. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inferencecited by 2
  11. 2025Global Resolution: Optimal Multi-Draft Speculative Sampling via Convex Minimizationcited by 2
  12. 2026FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibrationcited by 2
  13. 2026Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chainscited by 2
  14. 2026Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Draftingcited by 2
  15. 2026BlockPilot: Instance-Adaptive Policy Learning for Diffusion-based Speculative Decodingcited by 2
  16. 2026Unlocking Lossless Speedups in LLMs via Discrete Diffusioncited by 2
  17. 2026One-Token Verification for Reasoning Correctness Estimationcited by 1
  18. 2026Thinking into the Future: Latent Lookahead Training for Transformerscited by 1
  19. 2026Gumbel Distillation for Parallel Text Generationcited by 1
  20. 2026K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modelingcited by 1
  21. 2026H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cachecited by 1
  22. 2026Temporal Guidance for Large Language Modelscited by 0
  23. 2026Greedy Multi-Path Block Verification for Faster Decoding in Speculative Samplingcited by 0
  24. 2026Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decodingcited by 0
  25. 2026Efficient Training-Free Multi-Token Prediction via Embedding-Space Probingcited by 0
  26. 2026How Transformers Learn to Plan via Multi-Token Predictioncited by 0
  27. 2026Towards Long-Horizon Vessel Trajectory and Destination Forecasting with Reasoning Large Language Modelscited by 0
  28. 2026AdaMTP: An Adaptive Training Paradigm for Multi-Token Predictioncited by 0
  29. 2026LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimizationcited by 0
  30. 2026EchoRec: Multi-Item Prediction-Empowered Generative Recommendation via Cycle-Consistent Preference Alignmentcited by 0
  31. 2026Efficient Training with Foresight: Multi-Token Auxiliary Supervision for Autoregressive Image Generationcited by 0
  32. 2026SEED: Self-Speculative Decoding via Implicit Encoder-Decodercited by 0
  33. 2026BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Accelerationcited by 0
  34. 2026More Than Words: Compositional Tokenization for Efficient Language Modelscited by 0
  35. 2026DiffuPlex: Accelerating Full-Duplex Spoken Dialog Models via Rolling Masked Diffusioncited by 0