# Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix

arXiv: https://arxiv.org/abs/2607.06925 · HTML: https://mlsanity.com/paper/2607.06925/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 8 Jul 2026.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 18 of 18)

- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels](https://mlsanity.com/paper/2603.19312/) (2026)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models](https://mlsanity.com/paper/2210.05861/) (2022)
- [VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model](https://mlsanity.com/paper/2602.10098/) (2026)
- [Planning with Reasoning using Vision Language World Model](https://mlsanity.com/paper/2509.02722/) (2025)
- [Causal-JEPA: Learning World Models through Object-Level Latent Masking](https://mlsanity.com/paper/2602.11389/) (2026)
- [ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation](https://mlsanity.com/paper/2506.23126/) (2025)
- [SOLD: Slot Object-Centric Latent Dynamics Models for Relational Manipulation Learning from Pixels](https://mlsanity.com/paper/2410.08822/) (2024)
- [ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model](https://mlsanity.com/paper/2603.22281/) (2026)
- [Understanding-in-Generation: Reinforcing Generative Capability of Unified Model via Infusing Understanding into Generation](https://mlsanity.com/paper/2509.18639/) (2025)
- [Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representation](https://mlsanity.com/paper/2603.01549/) (2026)
- [Factored Latent Action World Models](https://mlsanity.com/paper/2602.16229/) (2026)
- [Vision-Language Model Predictive Control for Manipulation Planning and Trajectory Generation](https://mlsanity.com/paper/2504.05225/) (2025)
- [Terra: Explorable Native 3D World Model with Point Latents](https://mlsanity.com/paper/2510.14977/) (2025)

## Abstract (arXiv)

Compact world models that condition on a language goal promise to ground relations such as ``put the red block left of the blue block'' using a sparse set of explicit reference anchors. We ask when such references actually ground a relation, and identify a trap: a goal-conditioned predictor reaches a striking 0.90 relation-readout accuracy, yet this is instruction transcription, not perception. Withholding the goal collapses it to chance (0.90!→!0.27, three seeds) and a counterfactual instruction makes the predicted anchors follow the false instruction 94.5% of the time (true scene 2.3%; N=256). Tested across three settings and a within-task ablation, our central claim characterizes the confound: instruction leakage occurs when the scored quantity is transcribable from the instruction (when the instruction names the answer) and is essentially independent of how predictive the non-instruction inputs are. Our tabletop and the external BabyAI benchmark leak, whereas a Language-Table forward-dynamics world model whose instruction names referents does not, until the instruction is augmented to name the direction; and degrading the action never increases leakage, the opposite of what predictor-competition predicts. The diagnosis prescribes the fix: keep the goal out of the dynamics (it belongs to the planner's cost) and supervise the read path, recovering genuine, instruction-independent grounding (0.88, identical with and without the goal). The detection protocol and remedy apply to any goal-conditioned world model whose instruction names the scored quantity.