mlsanityData to 14 Oct 2026

Improving Vision-and-Language Navigation by Generating Future-View Image Semantics

arXiv 2304.04907 ↗First posted 11 Apr 2023cs.CV cs.AI cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

Vision-and-Language Navigation (VLN) is the task that requires an agent to navigate through the environment based on natural language instructions. At each step, the agent takes the next action by selecting from a set of navigable locations. In this paper, we aim to take one step further and explore whether the agent can benefit from generating the potential future view during navigation. Intuitively, humans will have an expectation of how the future environment will look like, based on the natural language instructions and surrounding views, which will aid correct navigation. Hence, to equip the agent with this ability to generate the semantics of future navigation views, we first propose three proxy tasks during the agent's in-domain pre-training: Masked Panorama Modeling (MPM), Masked Trajectory Modeling (MTM), and Action Prediction with Image Generation (APIG). These three objectives teach the model to predict missing views in a panorama (MPM), predict missing steps in the full trajectory (MTM), and generate the next view based on the full instruction and navigation history (APIG), respectively. We then fine-tune the agent on the VLN task with an auxiliary loss that minimizes the difference between the view semantics generated by the agent and the ground truth view semantics of the next step. Empirically, our VLN-SIG achieves the new state-of-the-art on both the Room-to-Room dataset and the CVDN dataset. We further show that our agent learns to fill in missing patches in future views qualitatively, which brings more interpretability over agents' predicted actions. Lastly, we demonstrate that learning to predict future view semantics also enables the agent to have better performance on longer paths.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 45 of its 48 references that are in the map. First posted on arXiv 11 Apr 2023.

Builds on

45 of 48 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  11. 2018On Evaluation of Embodied Navigation Agentscited by 547
  12. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
33 more
  1. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  2. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  3. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  4. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  5. 2019Vision-and-Dialog Navigationcited by 250
  6. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  7. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
  8. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  9. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  10. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  11. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 130
  12. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 130
  13. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 119
  14. 2021Pathdreamer: A World Model for Indoor Navigationcited by 118
  15. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 116
  16. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 105
  17. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  18. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  19. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  20. 2021Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environmentscited by 94
  21. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  22. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  23. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  24. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  25. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 70
  26. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 50
  27. 2022ADAPT: Vision-Language Navigation with Modality-Aligned Action Promptscited by 45
  28. 2020Environment-agnostic Multitask Learning for Natural Language Grounded Navigationcited by 43
  29. 2022VL-BEiT: Generative Vision-Language Pretrainingcited by 39
  30. 2021Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Informationcited by 31
  31. 2022MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrievalcited by 29
  32. 2020Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street Viewcited by 18
  33. 2019Multimodal Attention Networks for Low-Level Vision-and-Language Navigationcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 15; 2025: 18; 2026: 7.

46 papers.

Sort:

By area:

  1. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
  2. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 117
  3. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  4. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  5. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 65
  6. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  7. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 45
  8. 2025CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memorycited by 32
  9. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  10. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  11. 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 25
  12. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
34 more
  1. 2024GenEx: Generating an Explorable Worldcited by 21
  2. 2025Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Groundingcited by 18
  3. 2024OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and StructurEd Representationcited by 17
  4. 2024Correctable Landmark Discovery via Large Models for Vision-Language Navigationcited by 17
  5. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 17
  6. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  7. 2024Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigationcited by 15
  8. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 15
  9. 2024VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigationcited by 14
  10. 2025Do Visual Imaginations Improve Vision-and-Language Navigation Agents?cited by 14
  11. 2024NavHint: Vision and Language Navigation Agent with a Hint Generatorcited by 12
  12. 2024Generative World Explorercited by 12
  13. 2025REGNav: Room Expert Guided Image-Goal Navigationcited by 12
  14. 2024Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environmentscited by 10
  15. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  16. 2025CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigationcited by 5
  17. 2026textscNaVIDA: Vision-Language Navigation with Inverse Dynamics Augmentationcited by 5
  18. 2023Masked Path Modeling for Vision-and-Language Navigationcited by 4
  19. 2025EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoningcited by 4
  20. 2026Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigationcited by 4
  21. 2025Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigationcited by 3
  22. 2025UNeMo: Collaborative Visual-Language Reasoning and Navigation via a Multimodal World Modelcited by 3
  23. 2023Interactive Semantic Map Representation for Skill-based Visual Object Navigationcited by 2
  24. 2024NaVIP: An Image-Centric Indoor Navigation Solution for Visually Impaired Peoplecited by 2
  25. 2024World-Consistent Data Generation for Vision-and-Language Navigationcited by 2
  26. 2026WorldMAP: Bootstrapping Vision-Language Navigation Trajectory Prediction with Generative World Modelscited by 2
  27. 2026SpaAct: Spatially-Activated Transition Learning with Curriculum Adaptation for Vision-Language Navigationcited by 2
  28. 2025Fine-Grained Instruction-Guided Graph Reasoning for Vision-and-Language Navigationcited by 1
  29. 2025Harnessing Input-Adaptive Inference for Efficient VLNcited by 1
  30. 2025ST-Booster: An Iterative SpatioTemporal Perception Booster for Vision-and-Language Navigation in Continuous Environmentscited by 0
  31. 2025Walk and Read Less: Improving the Efficiency of Vision-and-Language Navigation via Tuning-Free Multimodal Token Pruningcited by 0
  32. 2026Implicit Geometry Representations for Vision-and-Language Navigation from Web Videoscited by 0
  33. 2026LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoningcited by 0
  34. 2026FINE: Future-Informed Navigation Encoding for Data-Efficient Vision-Language Navigationcited by 0