# Tracking Is Not Permanence: What Video World Models Keep of a Hidden Object

arXiv: https://arxiv.org/abs/2610.07355 · HTML: https://mlsanity.com/paper/2610.07355/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Video-generation world models and simulators, one of 11 directions in World models.
- Builds on 19 of its 52 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Placed in

- World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/)

## Builds on (most cited first, 19 of 19)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video](https://mlsanity.com/paper/2505.11709/) (2025)
- [WorldMem: Long-term Consistent World Simulation with Memory](https://mlsanity.com/paper/2504.12369/) (2025)
- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [Do generative video models understand physical principles?](https://mlsanity.com/paper/2501.09038/) (2025)
- [V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning](https://mlsanity.com/paper/2603.14482/) (2026)
- [Intuitive physics understanding emerges from self-supervised pretraining on natural videos](https://mlsanity.com/paper/2502.11831/) (2025)
- [Long-Context State-Space Video World Models](https://mlsanity.com/paper/2505.20171/) (2025)
- [IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments](https://mlsanity.com/paper/2506.09849/) (2025)
- [Interpreting Physics in Video World Models](https://mlsanity.com/paper/2602.07050/) (2026)
- [Unifying (Machine) Vision via Counterfactual World Modeling](https://mlsanity.com/paper/2306.01828/) (2023)
- [Current World Models Lack a Persistent State Core](https://mlsanity.com/paper/2606.20545/) (2026)
- [Occlusion resistant learning of intuitive physics from videos](https://mlsanity.com/paper/2005.00069/) (2020)
- [The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show](https://mlsanity.com/paper/2606.05328/) (2026)
- [How Do Video Foundation Models Encode Intuitive Physics? Probing Across Pretraining Paradigms](https://mlsanity.com/paper/2606.09646/) (2026)
- [Can Video World Models Track Unobserved World States?](https://mlsanity.com/paper/2608.30692/) (2026)
- [HERA: Historical Evidence Routing Adapter for Physical Prediction in Latent World Models](https://mlsanity.com/paper/2608.05523/) (2026)

## Abstract (arXiv)

Video world models track objects they can see; we ask what they keep of objects they cannot. We hide an object from a frozen V-JEPA 2 predictor and compare its prediction for the hidden region with the encoder's representation of two worlds that differ only inside that region. The predictor's decision keeps a stationary object in part and one carried inside a container not at all, and loses a moving one within 0.3 s (0.5 s under V-JEPA's own tube mask; ViT-H keeps it to 1.1 s at pretraining's 90% masking ratio); in projection a trace remains, below the midpoint, at 14-60% of what a baseline copying the last view retains. The information is there: the encoder reads the object's presence at 1.00 and keeps a closed container's contents decodable for 3.5 s, while the predictor's output, read with the encoder's own probe, contains the ball in 2% of scenes once the box has been closed for half a second. On rendered scenes, permanence is missing on the predictor's side, and training installs it cheaply as a prior: three thousand predictor-only steps on synthetic containers take this belief from 0.05 to 1.00 against two matched controls. They also raise IntPhys-2019 from 84.2% to 93.3%, but so does a curriculum without containers, and which training habit the benchmark credits changes with its scoring rule. Continued training with tube masks produces 1.1-1.6 s of moving-object carry-over on manipulation and internet-style video, so the deficit is not intrinsic to latent prediction. VideoMAE keeps almost nothing, and Cosmos's next-token prediction keeps a stationary hidden object but not one carried inside a moving container.