# CST-WM: A Causally Structured World Model for Embodied Visual Tracking arXiv: https://arxiv.org/abs/2609.06302 · HTML: https://mlsanity.com/paper/2609.06302/ ## Key facts - No citing papers in the map yet (data through 14 Oct 2026). - Placed in Embodied and robot world models, one of 11 directions in World models. - Builds on 39 of its 45 references that are in the map. - First posted on arXiv 5 Sep 2026. ## Placed in - World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/) ## Builds on (most cited first, 30 of 39) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024) - [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023) - [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024) - [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020) - [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019) - [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018) - [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023) - [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020) - [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018) - [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024) - [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023) - [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022) - [Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving](https://mlsanity.com/paper/2311.17918/) (2023) - [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019) - [Navigation World Models](https://mlsanity.com/paper/2412.03572/) (2024) - [DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving](https://mlsanity.com/paper/2309.09777/) (2023) - [Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots](https://mlsanity.com/paper/2310.13724/) (2023) - [Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks](https://mlsanity.com/paper/2412.06224/) (2024) - [PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators](https://mlsanity.com/paper/2406.20083/) (2024) - [Principles and Guidelines for Evaluating Social Robot Navigation Algorithms](https://mlsanity.com/paper/2306.16740/) (2023) - [TrackVLA: Embodied Visual Tracking in the Wild](https://mlsanity.com/paper/2505.23189/) (2025) - [Towards Distraction-Robust Active Visual Tracking](https://mlsanity.com/paper/2106.10110/) (2021) - [Offline Reinforcement Learning for Visual Navigation](https://mlsanity.com/paper/2212.08244/) (2022) - [Follow Anything: Open-set detection, tracking, and following in real-time](https://mlsanity.com/paper/2308.05737/) (2023) - [End-to-end Active Object Tracking via Reinforcement Learning](https://mlsanity.com/paper/1705.10561/) (2017) - [Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL](https://mlsanity.com/paper/2404.09857/) (2024) - [TrackVLA++: Unleashing Reasoning and Memory Capabilities in VLA Models for Embodied Visual Tracking](https://mlsanity.com/paper/2510.07134/) (2025) - [Pose-Assisted Multi-Camera Collaboration for Active Object Tracking](https://mlsanity.com/paper/2001.05161/) (2020) - [RSPT: Reconstruct Surroundings and Predict Trajectories for Generalizable Active Object Tracking](https://mlsanity.com/paper/2304.03623/) (2023) ## Abstract (arXiv) Embodied visual tracking requires a robot to choose actions that keep a moving target observable at a suitable distance, and to recover it after occlusion, out-of-view drift, or distractor crossings. We cast the task as planning over future target evidence with an action-conditioned world model. In logged tracking data, however, the behavior policy's actions are correlated with where the target is, so a generic predictor can learn a shortcut: it writes the current action directly into its prediction of target evidence, instead of letting the action affect that evidence only by moving the robot and changing what it observes. We call this failure causal hallucination; the resulting rollouts look plausible but rank candidate actions for the wrong reason. We propose CST-WM, a causally structured world model whose state is split into target-evidence, robot, and observation branches. Its transition removes the same-step edge from action to target evidence but keeps the path through robot motion and the resulting views, so candidate actions are still distinguished by their predicted ego-motion. With rollout-based model-predictive control, a single model handles both steady following and re-acquisition after target loss. On EVT-Bench and Habitat 3.0, covering standard tracking, target-loss recovery, and cross-dataset transfer, CST-WM improves following, distance-range control, safety, and re-acquisition over reactive trackers and world-model baselines, and removing the action mask causes the largest drop in re-acquisition among our ablations. Offline, CST-WM has lower multi-step rollout error, and its ranking of candidate actions agrees better with the simulator's. On a Unitree Go2 quadruped, CST-WM succeeds in 20 of 30 real-world trials under occlusion, distractor crossing, and fast motion, against 14 for TrackVLA.