mlsanityData to 14 Oct 2026

Do Visual Imaginations Improve Vision-and-Language Navigation Agents?

arXiv 2503.16394 ↗First posted 20 Mar 2025cs.CV cs.AI cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Vision-and-Language Navigation (VLN) agents are tasked with navigating an unseen environment using natural language instructions. In this work, we study if visual representations of sub-goals implied by the instructions can serve as navigational cues and lead to increased navigation performance. To synthesize these visual representations or imaginations, we leverage a text-to-image diffusion model on landmark references contained in segmented instructions. These imaginations are provided to VLN agents as an added modality to act as landmark cues and an auxiliary loss is added to explicitly encourage relating these with their corresponding referring expressions. Our findings reveal an increase in success rate (SR) of around 1 point and up to 0.5 points in success scaled by inverse path length (SPL) across agents. These results suggest that the proposed approach reinforces visual understanding compared to relying on language instructions alone. Code and data for our work can be found at https://www.akhilperincherry.com/VLN-Imagine-website/.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

36 of 47 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  6. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  8. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  9. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  10. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  11. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  12. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
24 more
  1. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  2. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  3. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  4. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 269
  5. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  6. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  7. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  8. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
  9. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 122
  10. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 117
  11. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 106
  12. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 105
  13. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 98
  14. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  15. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 66
  16. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  17. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 58
  18. 2022Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instancescited by 55
  19. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  20. 2022ADAPT: Vision-Language Navigation with Modality-Aligned Action Promptscited by 45
  21. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  22. 2023Can Pre-Trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning?cited by 20
  23. 2023Behavioral Analysis of Vision-and-Language Navigation Agentscited by 8
  24. 2023Does VLN Pretraining Work with Nonsensical or Irrelevant Instructions?cited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 44
  2. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  3. 2025VISTAv2: World Imagination for Indoor Vision-and-Language Navigationcited by 6
  4. 2025Expand Your SCOPE: Semantic Cognition over Potential-Based Exploration for Embodied Visual Navigationcited by 4
  5. 2026WorldMAP: Bootstrapping Vision-Language Navigation Trajectory Prediction with Generative World Modelscited by 2
  6. 2026SpaAct: Spatially-Activated Transition Learning with Curriculum Adaptation for Vision-Language Navigationcited by 2
  7. 2025Harnessing Input-Adaptive Inference for Efficient VLNcited by 1
  8. 2025Teaching RL Agents to Act Better: VLM as Action Advisor for Online Reinforcement Learningcited by 1
  9. 2025Schr"odinger's Navigator: Imagining an Ensemble of Futures for Zero-Shot Object Navigationcited by 1
  10. 2026Uncertainty-Aware Gaussian Map for Vision-Language Navigationcited by 1
  11. 2026Global Commander and Local Operative: A Dual-Agent Framework for Scene Navigationcited by 0
  12. 2026LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoningcited by 0
2 more
  1. 2026Towards Dual-Brain Minimal Sufficient Representation for Vision-Language Navigationcited by 0
  2. 2026Query, Align, and Distill: Navigation-Aware Cross-Modal Interaction for Efficient Vision-and-Language Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VISTA: Generative Visual Imagination for Vision-and-Language Navigation
  2. 2026What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior
  3. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions
  4. 2025Landmark-Guided Knowledge for Vision-and-Language Navigation
  5. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents
  6. 2025ImagineNav++: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
  7. 2025A Navigation Framework Utilizing Vision-Language Models
  8. 2022LOViS: Learning Orientation and Visual Signals for Vision and Language Navigation