mlsanityData to 7 Oct 2026

One Model, Two Physical Stories: Auditing Misalignment in Multi-Modal World Modeling

arXiv 2609.14833 ↗First posted 13 Sep 2026cs.AIcited by 0

On the map: World models → Latent predictive world models

Abstract (arXiv)

World models, systems that generate what happens next given current environmental conditions, are increasingly being implemented with multi-modal generation in mind. However, generating multiple modalities simultaneously, such as visual simulations alongside physical state predictions in the form of text, introduces the risk of cross-modal inconsistency. Tested separately, both outputs may look convincing while still disagreeing: a model can calculate that a ball should rebound in one modality, then generate no rebound in another modality, to say nothing of diverging from real-world dynamics entirely. In this work we focus on two failures explicitly: Internal misalignment, the disagreement between the world model's generated video and the same world model's prediction in a different modalities, and external misalignment the disagreement between the world model's generation and an analytic physical environment. We derive common contracts of event, magnitude, timing, and construct a physics grounded pipeline to make comparisons measurable in both external and internal settings. We then ask whether progressively supplying the model's own contract (the A ladder for the internal setting) or a corrected physical contract (the B ladder for the external setting) closes the respective gaps. Across four mechanisms and 20 settings, we find that while language answers all 22 text probes correctly with respect to the true environment, the neutral video is often in disagreement, suggesting that the current unified backbones may not be capable of correct reasoning, internal consistency, and external physical fidelity all at once.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 6 of its 14 references that are in the map. First posted on arXiv 13 Sep 2026.

Builds on

6 of 14 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Causal Physics Steering in Video World Models via Concept Activation Vectors23 May
  2. 2026Interpreting Physics in Video World Models4 Feb

Other references · 4

  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  3. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  4. 2026Quantifying the Gap between Understanding and Generation within Unified Multimodal Modelscited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Trinity of Consistency as a Defining Principle for General World Models
  2. 2026World Models' Last Exam in Physics
  3. 2026Event-Conditioned Diagnostics of Kinematic, Contact, and Object-Permanence Structure in Passive Object-State World Models
  4. 2026OneWorld: Learning Consistent Physics Across Actions in World Models
  5. 2026From Generative Engines to Actionable Simulators: The Imperative of Physical Grounding in World Models
  6. 2026From Generation to Simulation: How Far Are World Models from Being True Simulators?
  7. 2026Trimming the Long-Tail of Visual World Modeling Evaluation
  8. 2026WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Concepts