World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning
On the map: World models → World models for language agents
also in 2 more
Abstract (arXiv)
World models and multimodal large language models (MLLMs) provide complementary capabilities for predicting future outcomes from static visual observations. World models can generate concrete visual rollouts of possible futures, while MLLMs can reason abstractly over questions, goals, and rules. However, generated rollouts are stochastic and may be visually plausible but task-incorrect, making it necessary to determine when visual simulation is useful, whether a rollout is credible, and how it should influence the final answer. We formulate this problem as controlled concrete reasoning, where a model learns to invoke, verify, and integrate visual future simulation alongside abstract reasoning. To study this setting, we construct two human-verified benchmarks, VRQABench for controllable spatial lookahead and OpenWorldQA for open-domain physical prediction, and propose Privileged-Future On-Policy Self-Distillation (PF-OPSD). During training, PF-OPSD uses ground-truth future videos and answers only as teacher-side privileged context to evaluate on-policy concrete-reasoning trajectories, while the deployable student never observes true futures at test time. Experimental results show that PF-OPSD outperforms baseline by 10.6% and 10.9% on VRQABench and OpenWorldQA, respectively, while increasing robustness to noisy or conflicting rollouts. Our code and dataset are available at https://github.com/yczhou001/PF-OPSD.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 40 of its 68 references that are in the map. First posted on arXiv 2 Jun 2026.
Builds on
40 of 68 references.
Closest ancestors · 3
Same direction, newest first
- 2026Visual Generation Unlocks Human-Like Reasoning through Multimodal World Models27 Jan
- 2026Current Agents Fail to Leverage World Model as Tool for Foresight7 Jan
- 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoning16 Jul
Other references · 37
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
25 more
- 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
- 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
- 2023A Survey on Multimodal Large Language Modelscited by 909
- 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
- 2017A simple neural network module for relational reasoningcited by 346
- 2020Experience Grounds Languagecited by 242
- 2025Video models are zero-shot learners and reasonerscited by 220
- 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 206
- 2018Compositional Attention Networks for Machine Reasoningcited by 183
- 2019PHYRE: A New Benchmark for Physical Reasoningcited by 137
- 2024MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolutioncited by 134
- 2024Visual In-Context Learning for Large Vision-Language Modelscited by 128
- 2023Thread of Thought Unraveling Chaotic Contextscited by 72
- 2026Helios: Real Real-Time Long Video Generation Modelcited by 65
- 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 56
- 2023Large Language Models Are Not Strong Abstract Reasonerscited by 53
- 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 41
- 2025ViMo: A Generative Visual GUI World Model for App Agentscited by 35
- 2026MobileDreamer: Generative Sketch World Model for GUI Agentcited by 23
- 2025Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Taskscited by 22
- 2024Video Representation Learning with Joint-Embedding Predictive Architecturescited by 15
- 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
- 2025World Models Should Prioritize the Unification of Physical and Social Dynamicscited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026WorldCast: Exploring Intelligent Visual Generation in World State Prediction
- 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning
- 2026PhiZero: A World Model Built Around Physical Language
- 2026X-Foresight: A Joint Vision-Action Causal Forecasting Network via Predictive World Modeling
- 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents
- 2026Integrating Simulation and Chain-of-thought Reasoning in Multimodal-Language Models For Physical Reasoning
- 2024WorldGPT: Empowering LLM as Multimodal World Model
- 2025Planning with Reasoning using Vision Language World Model