# Action Forcing: Training World Models on Unsupervised Video by Recovering Underlying Egomotion Bases

arXiv: https://arxiv.org/abs/2609.30595 · HTML: https://mlsanity.com/paper/2609.30595/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 40 of its 44 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 30 of 40)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024)
- [Diffusion for World Modeling: Visual Details Matter in Atari](https://mlsanity.com/paper/2405.12399/) (2024)
- [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024)
- [Navigation World Models](https://mlsanity.com/paper/2412.03572/) (2024)
- [VideoPhy: Evaluating Physical Commonsense for Video Generation](https://mlsanity.com/paper/2406.03520/) (2024)
- [VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation](https://mlsanity.com/paper/2312.14867/) (2023)
- [World Simulation with Video Foundation Models for Physical AI](https://mlsanity.com/paper/2511.00062/) (2025)
- [VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation](https://mlsanity.com/paper/2406.15252/) (2024)
- [Matrix-game 2.0: An open-source real-time and streaming interactive world model](https://mlsanity.com/paper/2508.13009/) (2025)
- [GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2503.20523/) (2025)
- [Motion Prompting: Controlling Video Generation with Motion Trajectories](https://mlsanity.com/paper/2412.02700/) (2024)
- [WorldMem: Long-term Consistent World Simulation with Memory](https://mlsanity.com/paper/2504.12369/) (2025)
- [GameFactory: Creating New Games with Generative Interactive Videos](https://mlsanity.com/paper/2501.08325/) (2025)
- [CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos](https://mlsanity.com/paper/2410.11831/) (2024)
- [Learning to Act without Actions](https://mlsanity.com/paper/2312.10812/) (2023)
- [AdaWorld: Learning Adaptable World Models with Latent Actions](https://mlsanity.com/paper/2503.18938/) (2025)
- [CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Models](https://mlsanity.com/paper/2503.10592/) (2025)
- [MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft](https://mlsanity.com/paper/2504.08388/) (2025)
- [RELIC: Interactive Video World Model with Long-Horizon Memory](https://mlsanity.com/paper/2512.04040/) (2025)
- [Yume: An Interactive World Generation Model](https://mlsanity.com/paper/2507.17744/) (2025)
- [Vid2World: Crafting Video Diffusion Models to Interactive World Models](https://mlsanity.com/paper/2505.14357/) (2025)
- [Video-Bench: Human-Aligned Video Generation Benchmark](https://mlsanity.com/paper/2504.04907/) (2025)

## Abstract (arXiv)

Synchronised action annotations are needed to train controllable world models and these datasets remain elusive. Existing approaches make use of instrumented platforms with calibrated sensors, costly manual annotation, or latent-action models which lack grounding. We instead turn ordinary unlabelled video into action-supervised training data by recovering (without training) a data-derived egomotion basis. We track pixel displacements across frames and exploit the recurring coherent structure induced by egomotion to obtain grounded control signals directly. Using a method as simple as principal components analysis perform this, we find that the leading components provide signed, scalable, and composable throttle--yaw controls, although the method can recover only motion axes represented in the data. To prevent a high-capacity video DiT from exploiting pixel-level supervision, an online latent critic distils a frozen decoder--tracker--PCA (Principal Components Analysis) teacher without backpropagating through the decoder or tracker. Finally we critique the use of video generation metrics to evaluate WMs and introduce an example of an alternative, reference-free evaluation method. We measure textitcontrollability, textitplausibility, textitconjuring (creating objects out of thin air) and textitgeometric integrity, revealing failures that conventional video metrics miss. We show that most baselines follow familiar action directions but struggle to reverse or remain stationary. Our model handles both while retaining compositional control and generation quality. Despite backwards actions being less than 1% of our training data, we find that the model learns to reverse, scale its response linearly, and compose throttle with steering, all simply by learning through a grounded action space.