mlsanityData to 7 Oct 2026

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

arXiv 1811.10092 ↗First posted 25 Nov 2018cs.CV cs.AI cs.CLcited by 304

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Vision-language navigation (VLN) is the task of navigating an embodied agent to carry out natural language instructions inside real 3D environments. In this paper, we study how to address three critical challenges for this task: the cross-modal grounding, the ill-posed feedback, and the generalization problems. First, we propose a novel Reinforced Cross-Modal Matching (RCM) approach that enforces cross-modal grounding both locally and globally via reinforcement learning (RL). Particularly, a matching critic is used to provide an intrinsic reward to encourage global matching between instructions and trajectories, and a reasoning navigator is employed to perform cross-modal grounding in the local visual scene. Evaluation on a VLN benchmark dataset shows that our RCM model significantly outperforms previous methods by 10% on SPL and achieves the new state-of-the-art performance. To improve the generalizability of the learned policy, we further introduce a Self-Supervised Imitation Learning (SIL) method to explore unseen environments by imitating its own past, good decisions. We demonstrate that SIL can approximate a better and more efficient policy, which tremendously minimizes the success rate performance gap between seen and unseen environments (from 30.7% to 11.7%).

Key facts. Cited by 304 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 65 references that are in the map. First posted on arXiv 25 Nov 2018.

Builds on

45 of 65 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  10. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
33 more
  1. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  2. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  3. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  4. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  5. 2018On Evaluation of Embodied Navigation Agentscited by 526
  6. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  7. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  8. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  9. 2016Visual Dialogcited by 377
  10. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  11. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  12. 2017Embodied Question Answeringcited by 335
  13. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  14. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  15. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  16. 2016Segmentation from Natural Language Expressionscited by 257
  17. 2017Count-Based Exploration with Neural Density Modelscited by 255
  18. 2016Learning to Navigate in Complex Environmentscited by 192
  19. 2018Neural Approaches to Conversational AIcited by 164
  20. 2018Self-Imitation Learningcited by 135
  21. 2015Natural Language Object Retrievalcited by 124
  22. 2014From Captions to Visual Concepts and Backcited by 109
  23. 2018Visual Representations for Semantic Target Driven Navigationcited by 89
  24. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  25. 2017MINOS: Multimodal Indoor Simulator for Navigation in Complex Environmentscited by 66
  26. 2015Video Paragraph Captioning Using Hierarchical Recurrent Neural Networkscited by 54
  27. 2017Video Captioning via Hierarchical Reinforcement Learningcited by 47
  28. 2018No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytellingcited by 46
  29. 2017BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systemscited by 30
  30. 2018Shifting the Baseline: Single Modality Performance on Visual Navigation & QAcited by 30
  31. 2018Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioningcited by 21
  32. 2015Learning Models for Following Natural Language Directions in Unknown Environmentscited by 8
  33. 2018Turbo Learning for Captionbot and Drawingbotcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 2; 2020: 3; 2021: 60; 2022: 50; 2023: 64; 2024: 54; 2025: 42; 2026: 28.

304 papers.

Sort:

By area:

  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  2. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  3. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  4. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  5. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  6. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  7. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  8. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  9. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  10. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  11. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  12. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
38 more
  1. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  2. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  3. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  4. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  5. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  6. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  7. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  8. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  9. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  10. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  11. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  12. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  13. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  14. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  15. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  16. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  17. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  18. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  19. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  20. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  21. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  22. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  23. 2021Cross-Modal Progressive Comprehension for Referring Segmentationcited by 102
  24. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  25. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  26. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  27. 2021Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environmentscited by 87
  28. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  29. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
  30. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  31. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  32. 2022Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environmentscited by 80
  33. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  34. 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Interventioncited by 77
  35. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  36. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  37. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74
  38. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71

50 most cited of 304 All 304 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Language-guided Navigation via Cross-Modal Grounding and Alternate Adversarial Learning
  2. 2026Nipping the Drift in the Bud: Retrospective Rectification for Robust Vision-Language Navigation
  3. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning
  4. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld
  5. 2026AgentVLN: Towards Agentic Vision-and-Language Navigation
  6. 2026Language-Conditioned World Modeling for Visual Navigation
  7. 2026SpaAct: Spatially-Activated Transition Learning with Curriculum Adaptation for Vision-Language Navigation
  8. 2025VENTURA: Adapting Image Diffusion Models for Unified Task Conditioned Navigation