mlsanityData to 14 Oct 2026

Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement

arXiv 2604.06155 ↗First posted 7 Apr 2026cs.LG cs.AI cs.CLcited by 3

On the map: World models → World models for language agents

Abstract (arXiv)

Whether Large Language Models (LLMs) develop coherent internal world models remains a core debate. While conventional Next-Token Prediction (NTP) focuses on one-step-ahead supervision, Multi-Token Prediction (MTP) has shown promise in learning more structured representations. In this work, we provide a theoretical perspective analyzing the gradient inductive bias of MTP, supported by empirical evidence, showing that MTP promotes the convergence toward internal belief states by inducing representational contractivity via gradient coupling. However, we reveal that standard MTP often suffers from structural hallucinations, where discrete token supervision encourages illegal shortcuts in latent space that violate environmental constraints. To address this, we propose a novel method Latent Semantic Enhancement MTP (LSE-MTP), which anchors predictions to ground-truth hidden state trajectories. Experiments on synthetic graphs and real-world Manhattan Taxi Ride show that LSE-MTP effectively bridges the gap between discrete tokens and continuous state representations, enhancing representation alignment, reducing structural hallucinations, and improving robustness to perturbations.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 33 of its 53 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

33 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Better World Models Can Lead to Better Post-Training Performance3 Dec
  2. 2025Next-Latent Prediction Transformers Learn Compact World Models8 Nov
  3. 2024Evaluating the World Model Implicit in a Generative Model6 Jun
  4. 2023Reasoning with Language Model is Planning with World Model24 May
  5. 2022Variational Causal Dynamics: Discovering Modular World Models from Interventions22 Jun
  6. 2020Language Models are Few-Shot Learners28 May

Other references · 27

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2018World Modelscited by 1,513
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2020On Layer Normalization in the Transformer Architecturecited by 922
  7. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  8. 2018On Lazy Training in Differentiable Programmingcited by 675
  9. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  10. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  11. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  12. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
15 more
  1. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 222
  2. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 216
  3. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  4. 2024The pitfalls of next-token predictioncited by 160
  5. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 142
  6. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  7. 2016The Predictron: End-To-End Learning and Planningcited by 93
  8. 2025(How) Do Language Models Track State?cited by 34
  9. 2024Language Models Represent Beliefs of Self and Otherscited by 31
  10. 2023GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approachcited by 21
  11. 2023Can Language Models Understand Physical Concepts?cited by 19
  12. 2024Semformer: Transformer Language Models with Semantic Planningcited by 9
  13. 2025Understanding and Enhancing the Planning Capability of Language Models via Multi-Token Predictioncited by 6
  14. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  15. 2025Alternatives To Next Token Prediction In Text Generation -- A Surveycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026World Models as Group Actionscited by 3
  2. 2026From Tokens to States: LLMs as a Special Case of World Models and the Continuous Path Beyondcited by 0
  3. 2026Multi-Horizon Consistency as Geometry: When Latent Dynamics Contract, and When They Do Notcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Confidence: The Rhythms of Reasoning in Generative Models
  2. 2026AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction
  3. 2025Large Language Models Encode Semantics and Alignment in Linearly Separable Representations
  4. 2026Language Models Struggle to Use Representations Learned In-Context
  5. 2026Hierarchical Latent Prediction for Language Models
  6. 2026Reasoning with Latent Tokens in Diffusion Language Models
  7. 2025Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation
  8. 2026A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phases