# Language-guided Navigation via Cross-Modal Grounding and Alternate Adversarial Learning

arXiv: https://arxiv.org/abs/2011.10972 · HTML: https://mlsanity.com/paper/2011.10972/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2020.

## Built on by (most cited first, 22 of 22)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2211.02219/) (2022)
- [Vision-Language Navigation: A Survey and Taxonomy](https://mlsanity.com/paper/2108.11544/) (2021)
- [Fully and Weakly Supervised Referring Expression Segmentation with End-to-End Learning](https://mlsanity.com/paper/2212.10278/) (2022)
- [PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Grounding](https://mlsanity.com/paper/2208.05647/) (2022)
- [Agent-Centric Relation Graph for Object Visual Navigation](https://mlsanity.com/paper/2111.14422/) (2021)
- [CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation](https://mlsanity.com/paper/2103.00852/) (2021)
- [PASTS: Progress-Aware Spatio-Temporal Transformer Speaker For Vision-and-Language Navigation](https://mlsanity.com/paper/2305.11918/) (2023)
- [PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer](https://mlsanity.com/paper/2407.03813/) (2024)
- [Self-Training Vision Language BERTs with a Unified Conditional Model](https://mlsanity.com/paper/2201.02010/) (2022)
- [Cognition Transferring and Decoupling for Text-supervised Egocentric Semantic Segmentation](https://mlsanity.com/paper/2410.01341/) (2024)
- [Temporal-Enhanced Multimodal Transformer for Referring Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/2410.13437/) (2024)
- [NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigation](https://mlsanity.com/paper/2510.16457/) (2025)
- [MDFM: Multi-Decision Fusing Model for Few-Shot Learning](https://mlsanity.com/paper/2112.00690/) (2021)
- [NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularization](https://mlsanity.com/paper/2507.10894/) (2025)
- [Trajectory-Diversity-Driven Robust Vision-and-Language Navigation](https://mlsanity.com/paper/2603.15370/) (2026)
- [Referring Remote Sensing Image Segmentation with Cross-view Semantics Interaction Network](https://mlsanity.com/paper/2508.01331/) (2025)
- [Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency](https://mlsanity.com/paper/2603.09798/) (2026)
- [Unsupervised Visual Odometry and Action Integration for PointGoal Navigation in Indoor Environment](https://mlsanity.com/paper/2210.00413/) (2022)
- [EAVL: Explicitly Align Vision and Language for Referring Image Segmentation](https://mlsanity.com/paper/2308.09779/) (2023)
- [Query, Align, and Distill: Navigation-Aware Cross-Modal Interaction for Efficient Vision-and-Language Navigation](https://mlsanity.com/paper/2609.33097/) (2026)

## Abstract (arXiv)

The emerging vision-and-language navigation (VLN) problem aims at learning to navigate an agent to the target location in unseen photo-realistic environments according to the given language instruction. The main challenges of VLN arise mainly from two aspects: first, the agent needs to attend to the meaningful paragraphs of the language instruction corresponding to the dynamically-varying visual environments; second, during the training process, the agent usually imitate the shortest-path to the target location. Due to the discrepancy of action selection between training and inference, the agent solely on the basis of imitation learning does not perform well. Sampling the next action from its predicted probability distribution during the training process allows the agent to explore diverse routes from the environments, yielding higher success rates. Nevertheless, without being presented with the shortest navigation paths during the training process, the agent may arrive at the target location through an unexpected longer route. To overcome these challenges, we design a cross-modal grounding module, which is composed of two complementary attention mechanisms, to equip the agent with a better ability to track the correspondence between the textual and visual modalities. We then propose to recursively alternate the learning schemes of imitation and exploration to narrow the discrepancy between training and inference. We further exploit the advantages of both these two learning schemes via adversarial learning. Extensive experimental results on the Room-to-Room (R2R) benchmark dataset demonstrate that the proposed learning scheme is generalized and complementary to prior arts. Our method performs well against state-of-the-art approaches in terms of effectiveness and efficiency.