mlsanityData to 7 Oct 2026

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning

arXiv 2411.04983 ↗First posted 7 Nov 2024cs.RO cs.AIcited by 357

On the map: World models → Latent predictive world models

Abstract (arXiv)

The ability to predict future outcomes given control actions is fundamental for physical reasoning. However, such predictive models, often called world models, remains challenging to learn and are typically developed for task-specific solutions with online policy learning. To unlock world models' true potential, we argue that they should 1) be trainable on offline, pre-collected trajectories, 2) support test-time behavior optimization, and 3) facilitate task-agnostic reasoning. To this end, we present DINO World Model (DINO-WM), a new method to model visual dynamics without reconstructing the visual world. DINO-WM leverages spatial patch features pre-trained with DINOv2, enabling it to learn from offline behavioral trajectories by predicting future patch features. This allows DINO-WM to achieve observational goals through action sequence optimization, facilitating task-agnostic planning by treating goal features as prediction targets. We demonstrate that DINO-WM achieves zero-shot behavioral solutions at test time on six environments without expert demonstrations, reward modeling, or pre-learned inverse models, outperforming prior state-of-the-art work across diverse task families such as arbitrarily configured mazes, push manipulation with varied object shapes, and multi-particle scenarios.

Key facts. Cited by 357 papers in the map (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 42 of its 64 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

42 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture19 Jan
  2. 2022Temporal Difference Learning for Model Predictive Control9 Mar
  3. 2018Learning Latent Dynamics for Planning from Pixels12 Nov

Other references · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  9. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  10. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  11. 2018World Modelscited by 1,444
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
27 more
  1. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  2. 2023Mastering Diverse Domains through World Modelscited by 1,193
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  4. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  5. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  6. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  7. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  8. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  9. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  10. 2022Transformers are Sample-Efficient World Modelscited by 317
  11. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  12. 2022Masked Visual Pre-training for Motor Controlcited by 246
  13. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  14. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  15. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  16. 2024Behavior Generation with Latent Actionscited by 159
  17. 2023Structured World Models from Human Videoscited by 144
  18. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
  19. 2021Discovering and Achieving Goals via World Modelscited by 131
  20. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 131
  21. 2018Zero-Shot Visual Imitationcited by 114
  22. 2024BAKU: An Efficient Transformer for Multi-Task Policy Learningcited by 80
  23. 2019Learning to Manipulate Deformable Objects without Demonstrationscited by 76
  24. 2020Learning Predictive Representations for Deformable Objects Using Contrastive Estimationcited by 72
  25. 2024AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulationcited by 52
  26. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 43
  27. 2023ALAN: Autonomously Exploring Robotic Agents in the Real Worldcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 90; 2026: 265.

357 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  3. 2024Navigation World Modelscited by 262
  4. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelscited by 237
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  7. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  8. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  9. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 77
  10. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 65
  11. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  12. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 61
38 more
  1. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 56
  2. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  3. 2024DINO-Foresight: Looking into the Future with DINOcited by 44
  4. 2025Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysiscited by 44
  5. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  6. 2026Learning Latent Action World Models In The Wildcited by 42
  7. 2025DynaGuide: Steering Diffusion Polices with Active Dynamic Guidancecited by 39
  8. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 39
  9. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 38
  10. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 38
  11. 2026Solaris: Building a Multiplayer Video World Model in Minecraftcited by 38
  12. 2025What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?cited by 37
  13. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  14. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36
  15. 2026Hierarchical Planning with Latent World Modelscited by 34
  16. 2025Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Modelcited by 33
  17. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  18. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
  19. 2026Temporal Straightening for Latent Planningcited by 32
  20. 2025Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failurescited by 30
  21. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
  22. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 30
  23. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 27
  24. 2026When Does LeJEPA Learn a World Model?cited by 27
  25. 2025ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulationcited by 26
  26. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
  27. 2026Causal-JEPA: Learning World Models through Object-Level Latent Maskingcited by 25
  28. 2025LaDi-WM: A Latent Diffusion-based World Model for Predictive Manipulationcited by 24
  29. 2025Inference-Time Enhancement of Generative Robot Policies via Predictive World Modelingcited by 22
  30. 2025PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulationcited by 22
  31. 2025Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videoscited by 22
  32. 2026Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Modelscited by 21
  33. 2026LLM-as-a-Verifier: A General-Purpose Verification Frameworkcited by 21
  34. 2025WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planningcited by 19
  35. 2025Value-guided action planning with JEPA world modelscited by 19
  36. 2025SCALAR: Scale-wise Controllable Visual Autoregressive Learningcited by 18
  37. 2025Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interactioncited by 18
  38. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learningcited by 17

50 most cited of 357 All 357 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Representations for World-Action Modeling
  2. 2026OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining
  3. 2026VisualPatchWorld: Code World Models as Latent Structured Representations for Planning
  4. 2026Enhancing Policy Learning with World-Action Model
  5. 2026World Action Modeling with Progressive Visual Planning
  6. 2026HarnessWAM: Bridging Prediction and Deliberation in World Action Models
  7. 2025Zero-shot World Models via Search in Memory
  8. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction