mlsanityData to 7 Oct 2026

Pathdreamer: A World Model for Indoor Navigation

arXiv 2105.08756 ↗First posted 18 May 2021cs.CV cs.LGcited by 117

On the map: World models → World models for navigation

Abstract (arXiv)

People navigating in unfamiliar buildings take advantage of myriad visual, spatial and semantic cues to efficiently achieve their navigation goals. Towards equipping computational agents with similar capabilities, we introduce Pathdreamer, a visual world model for agents navigating in novel indoor environments. Given one or more previous visual observations, Pathdreamer generates plausible high-resolution 360 visual observations (RGB, semantic segmentation and depth) for viewpoints that have not been visited, in buildings not seen during training. In regions of high uncertainty (e.g. predicting around corners, imagining the contents of an unseen room), Pathdreamer can predict diverse scenes, allowing an agent to sample multiple realistic outcomes for a given trajectory. We demonstrate that Pathdreamer encodes useful and accessible visual, spatial and semantic knowledge about human environments by using it in the downstream task of Vision-and-Language Navigation (VLN). Specifically, we show that planning ahead with Pathdreamer brings about half the benefit of looking ahead at actual observations from unobserved parts of the environment. We hope that Pathdreamer will help unlock model-based approaches to challenging embodied navigation tasks such as navigating to specified objects and VLN.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Placed in World models for navigation, one of 11 directions in World models. Builds on 77 of its 87 references that are in the map. First posted on arXiv 18 May 2021.

Builds on

77 of 87 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019Habitat: A Platform for Embodied AI Research2 Apr

Other references · 76

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  4. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  8. 2018World Modelscited by 1,444
  9. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  10. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  11. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  12. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
38 more
  1. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  2. 2020Mastering Atari with Discrete World Modelscited by 990
  3. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  4. 2018On Evaluation of Embodied Navigation Agentscited by 526
  5. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
  6. 2019Model-Based Reinforcement Learning for Ataricited by 495
  7. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  8. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  9. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  10. 2018Video-to-Video Synthesiscited by 388
  11. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  12. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  13. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  14. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  15. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  16. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  17. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  18. 2019DeepView: View Synthesis with Learned Gradient Descentcited by 253
  19. 2019Vision-and-Dialog Navigationcited by 246
  20. 2018Stochastic Video Generation with a Learned Priorcited by 244
  21. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  22. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  23. 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 239
  24. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  25. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  26. 2018Stereo Magnification: Learning View Synthesis using Multiplane Imagescited by 206
  27. 20203D Photography using Context-aware Layered Depth Inpaintingcited by 206
  28. 2017Learning a Multi-View Stereo Machinecited by 194
  29. 2019Pushing the Boundaries of View Extrapolation with Multiplane Imagescited by 191
  30. 2015DeepStereo: Learning to Predict New Views from the World's Imagerycited by 185
  31. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  32. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  33. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  34. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  35. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 154
  36. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  37. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  38. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 21; 2023: 18; 2024: 21; 2025: 30; 2026: 26.

117 papers.

Sort:

By area:

  1. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  2. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  3. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  4. 2024Navigation World Modelscited by 262
  5. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  6. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  7. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  8. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  9. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  10. 2022InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Imagescited by 82
  11. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  12. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
38 more
  1. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  2. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  3. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  4. 2024MultiDiff: Consistent Novel View Synthesis from a Single Imagecited by 67
  5. 2022Removing Objects From Neural Radiance Fieldscited by 66
  6. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
  7. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  8. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 63
  9. 2023NeRFiller: Completing Scenes via Generative 3D Inpaintingcited by 62
  10. 2024Advances in 3D Generation: A Surveycited by 62
  11. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  12. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  13. 2022Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Imagecited by 55
  14. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  15. 2022DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Modelscited by 51
  16. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  17. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 49
  18. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  19. 2022Retrospectives on the Embodied AI Workshopcited by 46
  20. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  21. 2023Persistent Nature: A Generative Model of Unbounded 3D Worldscited by 44
  22. 2025World-in-World: World Models in a Closed-Loop Worldcited by 44
  23. 2021Language Grounding with 3D Objectscited by 43
  24. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  25. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  26. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 41
  27. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  28. 2022Deep Learning for Omnidirectional Vision: A Survey and New Perspectivescited by 38
  29. 2022Simple and Effective Synthesis of Indoor 3D Scenescited by 33
  30. 2025CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memorycited by 30
  31. 20243D-SceneDreamer: Text-Driven 3D-Consistent Scene Generationcited by 29
  32. 2025Learning 3D Persistent Embodied World Modelscited by 29
  33. 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 25
  34. 2022Discovering Transferable Forensic Features for CNN-generated Images Detectioncited by 24
  35. 2024MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware Diffusion and Iterative Refinementcited by 24
  36. 20253D Scene Generation: A Surveycited by 24
  37. 2022360FusionNeRF: Panoramic Neural Radiance Fields with Joint Guidancecited by 23
  38. 2022EgoEnv: Human-centric environment representations from egocentric videocited by 22

50 most cited of 117 All 117 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026A 3D Isovist World Model -- Revealing a City's Unseen Geometry and Its Emergent Cross-City Signature
  2. 2025Imaginative World Modeling with Scene Graphs for Embodied Agent Navigation
  3. 2026SceneTeract: Probing and Improving Agent-Aware Activity Reasoning in 3D Indoor Scenes
  4. 2026Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation
  5. 2026Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation
  6. 2025NavA3: Understanding Any Instruction, Navigating Anywhere, Finding Anything
  7. 2026SpaceVLN: A Zero-Shot Vision-and-Language Navigation Agent with Online Spatial Cognitive Memory and Reasoning
  8. 2023Reasoning about the Unseen for Efficient Outdoor Object Navigation