mlsanityData to 7 Oct 2026

Navigation World Models

arXiv 2412.03572 ↗First posted 4 Dec 2024cs.CV cs.AI cs.LGcited by 262

On the map: World models → World models for navigation

Abstract (arXiv)

Navigation is a fundamental skill of agents with visual-motor capabilities. We introduce a Navigation World Model (NWM), a controllable video generation model that predicts future visual observations based on past observations and navigation actions. To capture complex environment dynamics, NWM employs a Conditional Diffusion Transformer (CDiT), trained on a diverse collection of egocentric videos of both human and robotic agents, and scaled up to 1 billion parameters. In familiar environments, NWM can plan navigation trajectories by simulating them and evaluating whether they achieve the desired goal. Unlike supervised navigation policies with fixed behavior, NWM can dynamically incorporate constraints during planning. Experiments demonstrate its effectiveness in planning trajectories from scratch or by ranking trajectories sampled from an external policy. Furthermore, NWM leverages its learned visual priors to imagine trajectories in unfamiliar environments from a single input image, making it a flexible and powerful tool for next-generation navigation systems.

Key facts. Cited by 262 papers in the map (data through 7 Oct 2026). Placed in World models for navigation, one of 11 directions in World models. Builds on 63 of its 74 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

63 of 74 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Pathdreamer: A World Model for Indoor Navigation18 May
  2. 2019Habitat: A Platform for Embodied AI Research2 Apr

Other references · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2016Layer Normalizationcited by 5,579
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
38 more
  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  3. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  4. 2018World Modelscited by 1,444
  5. 2020Mastering Atari with Discrete World Modelscited by 990
  6. 2024Genie: Generative Interactive Environmentscited by 706
  7. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  8. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  9. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  10. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  11. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  12. 2023Learning Interactive Real-World Simulatorscited by 440
  13. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  14. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  15. 2022MAGVIT: Masked Generative Video Transformercited by 341
  16. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  17. 2020Learning to Explore using Active Neural SLAMcited by 308
  18. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  19. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  20. 2024Diffusion Models Are Real-Time Game Enginescited by 282
  21. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  22. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  23. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  24. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 234
  25. 2019Understanding and Improving Layer Normalizationcited by 213
  26. 2023ViNT: A Foundation Model for Visual Navigationcited by 210
  27. 2017VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learningcited by 199
  28. 2016Learning to Navigate in Complex Environmentscited by 192
  29. 2022Masked World Models for Visual Controlcited by 169
  30. 2022GNM: A General Navigation Model to Drive Any Robotcited by 161
  31. 2022Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigationcited by 122
  32. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  33. 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 116
  34. 2018Zero-Shot Visual Imitationcited by 114
  35. 2019Learning Exploration Policies for Navigationcited by 113
  36. 2024Phased Consistency Modelscited by 83
  37. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 76
  38. 2023Learning to Model the World with Languagecited by 66

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 111; 2026: 150.

262 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  3. 2026Advancing Open-source World Modelscited by 144
  4. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  5. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  6. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  7. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  8. 2025Cameras as Relative Positional Encodingcited by 94
  9. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 86
  10. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  11. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 67
  12. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
38 more
  1. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
  2. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  3. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  4. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 49
  5. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 49
  6. 2025Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraftcited by 46
  7. 2025World-in-World: World Models in a Closed-Loop Worldcited by 44
  8. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 43
  9. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  10. 2026Learning Latent Action World Models In The Wildcited by 42
  11. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  12. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 41
  13. 2026Solaris: Building a Multiplayer Video World Model in Minecraftcited by 38
  14. 2025VRAG: Learning World Models for Interactive Video Generationcited by 37
  15. 2025What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?cited by 37
  16. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 36
  17. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  18. 2025World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulationcited by 35
  19. 2026Do World Action Models Generalize Better than VLAs? A Robustness Studycited by 35
  20. 2026Hierarchical Planning with Latent World Modelscited by 34
  21. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  22. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
  23. 2026Temporal Straightening for Latent Planningcited by 32
  24. 2025Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planningcited by 31
  25. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
  26. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 30
  27. 2025Astra: General Interactive World Model with Autoregressive Denoisingcited by 30
  28. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  29. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  30. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
  31. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  32. 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 24
  33. 2026Improved Baselines with Representation Autoencoderscited by 24
  34. 2025A Survey of Interactive Generative Videocited by 23
  35. 2025What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewardscited by 23
  36. 2025Inference-Time Enhancement of Generative Robot Policies via Predictive World Modelingcited by 22
  37. 2026LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Modelscited by 20
  38. 2026Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Modelscited by 20

50 most cited of 262 All 262 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026UniNav: A Unified World-Action Diffusion Model for Visual Navigation
  2. 2026WAM-Nav: Asymmetric Latent World-Action Modeling for Unified Visual Navigation
  3. 2022Tracking and Planning with Spatial World Models
  4. 2026Traj-LeWM: Path-Aware World-Model Planning via Latent Trajectory Cost
  5. 2026An Efficient and Multi-Modal Navigation System with One-Step World Model
  6. 2026Hydra: A Navigation World Action Model with Discrete Latent Planning and Continuous Flow-Matching Execution
  7. 2026EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields
  8. 2026Foresight Without Seeing: Latent Futures for World Action Models