# DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning

arXiv: https://arxiv.org/abs/2411.04983 · HTML: https://mlsanity.com/paper/2411.04983/

## Key facts

- Cited by 386 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 42 of its 64 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Planning to Explore via Self-Supervised World Models](https://mlsanity.com/paper/2005.05960/) (2020)
- [Transformers are Sample-Efficient World Models](https://mlsanity.com/paper/2209.00588/) (2022)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Masked Visual Pre-training for Motor Control](https://mlsanity.com/paper/2203.06173/) (2022)
- [Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control](https://mlsanity.com/paper/1812.00568/) (2018)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [Learning to Act from Actionless Videos through Dense Correspondences](https://mlsanity.com/paper/2310.08576/) (2023)

## Built on by (most cited first, 30 of 386)

- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [UniVLA: Learning to Act Anywhere with Task-centric Latent Actions](https://mlsanity.com/paper/2505.06111/) (2025)
- [LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels](https://mlsanity.com/paper/2603.19312/) (2026)
- [Navigation World Models](https://mlsanity.com/paper/2412.03572/) (2024)
- [DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos](https://mlsanity.com/paper/2602.06949/) (2026)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [AdaWorld: Learning Adaptable World Models with Latent Actions](https://mlsanity.com/paper/2503.18938/) (2025)
- [FLARE: Robot Learning with Implicit World Modeling](https://mlsanity.com/paper/2505.15659/) (2025)
- [Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models](https://mlsanity.com/paper/2502.14819/) (2025)
- [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025)
- [VL-JEPA: Joint Embedding Predictive Architecture for Vision-language](https://mlsanity.com/paper/2512.10942/) (2025)
- [VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators](https://mlsanity.com/paper/2510.00406/) (2025)
- [Back to the Features: DINO as a Foundation for Video World Models](https://mlsanity.com/paper/2507.19468/) (2025)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [Learning Latent Action World Models In The Wild](https://mlsanity.com/paper/2601.05230/) (2026)
- [Solaris: Building a Multiplayer Video World Model in Minecraft](https://mlsanity.com/paper/2602.22208/) (2026)
- [DINO-Foresight: Looking into the Future with DINO](https://mlsanity.com/paper/2412.11673/) (2024)
- [Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysis](https://mlsanity.com/paper/2502.00935/) (2025)
- [ReSim: Reliable World Simulation for Autonomous Driving](https://mlsanity.com/paper/2506.09981/) (2025)
- [DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance](https://mlsanity.com/paper/2506.13922/) (2025)
- [What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?](https://mlsanity.com/paper/2512.24497/) (2025)
- [Hierarchical Planning with Latent World Models](https://mlsanity.com/paper/2604.03208/) (2026)
- [RLVR-World: Training World Models with Reinforcement Learning](https://mlsanity.com/paper/2505.13934/) (2025)
- [NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos](https://mlsanity.com/paper/2510.08568/) (2025)
- [RISE: Self-Improving Robot Policy with Compositional World Model](https://mlsanity.com/paper/2602.11075/) (2026)
- [Temporal Straightening for Latent Planning](https://mlsanity.com/paper/2603.12231/) (2026)
- [Beyond Language Modeling: An Exploration of Multimodal Pretraining](https://mlsanity.com/paper/2603.03276/) (2026)
- [Planning with Reasoning using Vision Language World Model](https://mlsanity.com/paper/2509.02722/) (2025)
- [Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model](https://mlsanity.com/paper/2510.27607/) (2025)
- [PAN: A World Model for General, Actionable, and Long-Horizon World Simulation](https://mlsanity.com/paper/2511.09057/) (2025)

## Abstract (arXiv)

The ability to predict future outcomes given control actions is fundamental for physical reasoning. However, such predictive models, often called world models, remains challenging to learn and are typically developed for task-specific solutions with online policy learning. To unlock world models' true potential, we argue that they should 1) be trainable on offline, pre-collected trajectories, 2) support test-time behavior optimization, and 3) facilitate task-agnostic reasoning. To this end, we present DINO World Model (DINO-WM), a new method to model visual dynamics without reconstructing the visual world. DINO-WM leverages spatial patch features pre-trained with DINOv2, enabling it to learn from offline behavioral trajectories by predicting future patch features. This allows DINO-WM to achieve observational goals through action sequence optimization, facilitating task-agnostic planning by treating goal features as prediction targets. We demonstrate that DINO-WM achieves zero-shot behavioral solutions at test time on six environments without expert demonstrations, reward modeling, or pre-learned inverse models, outperforming prior state-of-the-art work across diverse task families such as arbitrarily configured mazes, push manipulation with varied object shapes, and multi-particle scenarios.