mlsanityData to 14 Oct 2026

DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation

arXiv 2308.07498 ↗First posted 14 Aug 2023cs.CV cs.AIcited by 85

On the map: World models → World models for navigation

Abstract (arXiv)

VLN-CE is a recently released embodied task, where AI agents need to navigate a freely traversable environment to reach a distant target location, given language instructions. It poses great challenges due to the huge space of possible strategies. Driven by the belief that the ability to anticipate the consequences of future actions is crucial for the emergence of intelligent and interpretable planning behavior, we propose DREAMWALKER -- a world model based VLN-CE agent. The world model is built to summarize the visual, topological, and dynamic properties of the complicated continuous environment into a discrete, structured, and compact representation. DREAMWALKER can simulate and evaluate possible plans entirely in such internal abstract world, before executing costly actions. As opposed to existing model-free VLN-CE agents simply making greedy decisions in the real world, which easily results in shortsighted behaviors, DREAMWALKER is able to make strategic planning through large amounts of ``mental experiments.'' Moreover, the imagined future scenarios reflect our agent's intention, making its decision-making process more transparent. Extensive experiments and ablation studies on VLN-CE dataset confirm the effectiveness of the proposed approach and outline fruitful directions for future work.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Placed in World models for navigation, one of 11 directions in World models. Builds on 80 of its 97 references that are in the map. First posted on arXiv 14 Aug 2023.

Builds on

80 of 97 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Pathdreamer: A World Model for Indoor Navigation18 May
  2. 2019Habitat: A Platform for Embodied AI Research2 Apr

Other references · 78

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Graph Attention Networkscited by 8,923
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  8. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  9. 2020Mastering Atari with Discrete World Modelscited by 1,007
  10. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  11. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  12. 2018On Evaluation of Embodied Navigation Agentscited by 547
38 more
  1. 2019Model-Based Reinforcement Learning for Ataricited by 502
  2. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  3. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  4. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  5. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  6. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  7. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  8. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  9. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  10. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
  11. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  12. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
  13. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  14. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  15. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  16. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  17. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  18. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  19. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  20. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  21. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 179
  22. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  23. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 176
  24. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
  25. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 169
  26. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 156
  27. 2021Episodic Transformer for Vision-and-Language Navigationcited by 156
  28. 2021Vision-Only Robot Navigation in a Neural Radiance Worldcited by 155
  29. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 140
  30. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 132
  31. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 130
  32. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 129
  33. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 105
  34. 2020Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learningcited by 103
  35. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  36. 2021Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environmentscited by 94
  37. 2019Exploring Model-based Planning with Policy Networkscited by 90
  38. 2022Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environmentscited by 86

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 16; 2025: 30; 2026: 37.

85 papers.

Sort:

By area:

  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  2. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 211
  3. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 118
  4. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 89
  5. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  6. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 65
  7. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  8. 2025World-in-World: World Models in a Closed-Loop Worldcited by 47
  9. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 45
  10. 2024Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigationcited by 39
  11. 2025CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memorycited by 32
  12. 2025CorrectNav: Self-Correction Flywheel Empowers Vision-Language-Action Navigation Modelcited by 27
38 more
  1. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  2. 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 25
  3. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 22
  4. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 21
  5. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 21
  6. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 20
  7. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  8. 2024PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulationcited by 17
  9. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 17
  10. 2025DreamNav: A Trajectory-Based Imaginative Framework for Zero-Shot Vision-and-Language Navigationcited by 17
  11. 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
  12. 2025NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Predictioncited by 16
  13. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 15
  14. 2025MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robotscited by 15
  15. 2025ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigationcited by 13
  16. 2025Aerial World Model for Long-horizon Visual Generation and Navigation in 3D Spacecited by 12
  17. 2025GC-VLN: Instruction as Graph Constraints for Training-free Vision-and-Language Navigationcited by 10
  18. 2026AwareVLN: Reasoning with Self-awareness for Vision-Language Navigationcited by 10
  19. 2024Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomalycited by 9
  20. 2026AgentVLN: Towards Agentic Vision-and-Language Navigationcited by 9
  21. 2024Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigationcited by 8
  22. 2025VLN-Zero: Rapid Exploration and Cache-Enabled Neurosymbolic Vision-Language Planning for Zero-Shot Transfer in Robot Navigationcited by 8
  23. 2025Aerial Vision-Language Navigation with a Unified Framework for Spatial, Temporal and Embodied Reasoningcited by 8
  24. 2026Dual-Anchoring: Addressing State Drift in Vision-Language Navigationcited by 7
  25. 2024Vision-Language Navigation with Continual Learningcited by 6
  26. 2025View Invariant Learning for Vision-Language Navigation in Continuous Environmentscited by 6
  27. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  28. 2025History-Enhanced Two-Stage Transformer for Aerial Vision-and-Language Navigationcited by 6
  29. 2026Graph World Models: Concepts, Taxonomy, and Future Directionscited by 6
  30. 2026ABot-N1: Toward a General Visual Language Navigation Foundation Modelcited by 6
  31. 2026World-Action Models for Robot Learning and Control: A Surveycited by 6
  32. 2025Other Vehicle Trajectories Are Also Needed: A Driving World Model Unifies Ego-Other Vehicle Trajectories in Video Latent Spacecited by 5
  33. 2025Interaction-Merged Motion Planning: Effectively Leveraging Diverse Motion Datasets for Robust Planningcited by 5
  34. 2026textscNaVIDA: Vision-Language Navigation with Inverse Dynamics Augmentationcited by 5
  35. 2026Your Vision-Language-Action Model Already Has Attention Heads For Path Deviation Detectioncited by 5
  36. 2026PanoWorld: Towards Spatial Supersensing in 360^circ Panorama Worldcited by 5
  37. 20263D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigationcited by 4
  38. 2025Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigationcited by 3

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation
  2. 2025A Navigation Framework Utilizing Vision-Language Models
  3. 2025WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigation
  4. 2026Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?
  5. 2025ETP-R1: Evolving Topological Planning with Reinforcement Fine-tuning for Vision-Language Navigation in Continuous Environments
  6. 2025Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning
  7. 2025Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresight
  8. 2025Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models