mlsanityData to 7 Oct 2026

Better & Faster Large Language Models via Multi-token Prediction

arXiv 2404.19737 ↗First posted 30 Apr 2024cs.CLcited by 344

Not on the map. Why?

Cited by 85 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models such as GPT and Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict multiple future tokens at once results in higher sample efficiency. More specifically, at each position in the training corpus, we ask the model to predict the following n tokens using n independent output heads, operating on top of a shared model trunk. Considering multi-token prediction as an auxiliary training task, we measure improved downstream capabilities with no overhead in training time for both code and natural language models. The method is increasingly useful for larger model sizes, and keeps its appeal when training for multiple epochs. Gains are especially pronounced on generative benchmarks like coding, where our models consistently outperform strong baselines by several percentage points. Our 13B parameter models solves 12 % more problems on HumanEval and 17 % more on MBPP than comparable next-token models. Experiments on small algorithmic tasks demonstrate that multi-token prediction is favorable for the development of induction heads and algorithmic reasoning capabilities. As an additional benefit, models trained with 4-token prediction are up to 3 times faster at inference, even with large batch sizes.

Key facts. Cited by 344 papers in the map (data through 7 Oct 2026). Builds on 34 of its 54 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

34 of 54 references.

References · 34

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
22 more
  1. 2022In-context Learning and Induction Headscited by 930
  2. 2021Measuring Coding Challenge Competence With APPScited by 879
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  4. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  5. 2022High Fidelity Neural Audio Compressioncited by 673
  6. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  7. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  8. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  9. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  10. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  11. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  12. 2022UL2: Unifying Language Learning Paradigmscited by 292
  13. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  14. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  15. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  16. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 116
  17. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 72
  18. 2019Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answeringcited by 58
  19. 2020BARThez: a Skilled Pretrained French Sequence-to-Sequence Modelcited by 39
  20. 2019Classifier Chains: A Review and Perspectivescited by 23
  21. 2018Multi-Target Prediction: A Unifying View on Problems and Methodscited by 15
  22. 2020Leveraging ParsBERT and Pretrained mT5 for Persian Abstractive Text Summarizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 48; 2025: 129; 2026: 167.

344 papers.

Sort:

By area:

  1. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  2. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  3. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  4. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  5. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  6. 2026MiMo-V2-Flash Technical Reportcited by 163
  7. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  8. 2025Training Agents Inside of Scalable World Modelscited by 122
  9. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
  10. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 100
  11. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  12. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
38 more
  1. 2025A Survey on Diffusion Language Modelscited by 75
  2. 2024Transformers represent belief state geometry in their residual streamcited by 73
  3. 2025OmniSVG: A Unified Scalable Vector Graphics Generation Modelcited by 71
  4. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  5. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
  6. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  7. 2025SuperBPE: Space Travel for Language Modelscited by 58
  8. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  9. 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 55
  10. 2025LongCat-Flash Technical Reportcited by 54
  11. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 54
  12. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  13. 2026The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligencecited by 50
  14. 2024BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Predictioncited by 49
  15. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  16. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  17. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 42
  18. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  19. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 38
  20. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  21. 2025TiDAR: Think in Diffusion, Talk in Autoregressioncited by 35
  22. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 34
  23. 2025LLM Pretraining with Continuous Conceptscited by 33
  24. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 31
  25. 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agentscited by 30
  26. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 29
  27. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 28
  28. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
  29. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  30. 2025Next Block Prediction: Video Generation via Semi-Autoregressive Modelingcited by 26
  31. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  32. 2026Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scalingcited by 24
  33. 2026Discrete Flow Mapscited by 23
  34. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  35. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  36. 2025Set Block Decoding is a Language Model Inference Acceleratorcited by 22
  37. 2025Diffusion LLM with Native Variable Generation Lengths: Let [EOS] Lead the Waycited by 22
  38. 2024Customize Your Visual Autoregressive Recipe with Set Autoregressive Modelingcited by 21

50 most cited of 344 All 344 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Distribution-Aware Companding Quantization of Large Language Models
  2. 2025On multi-token prediction for efficient LLM inference
  3. 2025Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
  4. 2025Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?
  5. 2026Stepwise Token Selection for Efficient Multimodal Large Language Models
  6. 2026Multi-Token Prediction via Self-Distillation
  7. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
  8. 2025Length-MAX Tokenizer for Language Models