# Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling

arXiv: https://arxiv.org/abs/1911.07308 · HTML: https://mlsanity.com/paper/1911.07308/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2019.

## Built on by (most cited first, 22 of 22)

- [Pathdreamer: A World Model for Indoor Navigation](https://mlsanity.com/paper/2105.08756/) (2021)
- [Structured Scene Memory for Vision-Language Navigation](https://mlsanity.com/paper/2103.03454/) (2021)
- [NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning](https://mlsanity.com/paper/2403.07376/) (2024)
- [The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigation](https://mlsanity.com/paper/2104.04167/) (2021)
- [Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigation](https://mlsanity.com/paper/2203.16586/) (2022)
- [Openfly: A comprehensive platform for aerial vision-language navigation](https://mlsanity.com/paper/2502.18041/) (2025)
- [On the Evaluation of Vision-and-Language Navigation Instructions](https://mlsanity.com/paper/2101.10504/) (2021)
- [Lana: A Language-Capable Navigator for Instruction Following and Generation](https://mlsanity.com/paper/2303.08409/) (2023)
- [Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectives](https://mlsanity.com/paper/2307.16851/) (2023)
- [Towards Versatile Embodied Navigation](https://mlsanity.com/paper/2210.16822/) (2022)
- [RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigation](https://mlsanity.com/paper/2412.08591/) (2024)
- [Top-N Recommendation with Counterfactual User Preference Simulation](https://mlsanity.com/paper/2109.02444/) (2021)
- [Correctable Landmark Discovery via Large Models for Vision-Language Navigation](https://mlsanity.com/paper/2405.18721/) (2024)
- [Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation](https://mlsanity.com/paper/2107.11252/) (2021)
- [CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations](https://mlsanity.com/paper/2207.02185/) (2022)
- [FedVLN: Privacy-preserving Federated Vision-and-Language Navigation](https://mlsanity.com/paper/2203.14936/) (2022)
- [EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoning](https://mlsanity.com/paper/2506.01551/) (2025)
- [Comparing Visual Reasoning in Humans and AI](https://mlsanity.com/paper/2104.14102/) (2021)
- [Deep Learning for Embodied Vision Navigation: A Survey](https://mlsanity.com/paper/2108.04097/) (2021)
- [Multimodal Sequential Generative Models for Semi-Supervised Language Instruction Following](https://mlsanity.com/paper/2301.00676/) (2022)
- [Accessible Instruction-Following Agent](https://mlsanity.com/paper/2305.06358/) (2023)
- [Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos](https://mlsanity.com/paper/2603.09259/) (2026)

## Abstract (arXiv)

Vision-and-Language Navigation (VLN) is a task where agents must decide how to move through a 3D environment to reach a goal by grounding natural language instructions to the visual surroundings. One of the problems of the VLN task is data scarcity since it is difficult to collect enough navigation paths with human-annotated instructions for interactive environments. In this paper, we explore the use of counterfactual thinking as a human-inspired data augmentation method that results in robust models. Counterfactual thinking is a concept that describes the human propensity to create possible alternatives to life events that have already occurred. We propose an adversarial-driven counterfactual reasoning model that can consider effective conditions instead of low-quality augmented data. In particular, we present a model-agnostic adversarial path sampler (APS) that learns to sample challenging paths that force the navigator to improve based on the navigation performance. APS also serves to do pre-exploration of unseen environments to strengthen the model's ability to generalize. We evaluate the influence of APS on the performance of different VLN baseline models using the room-to-room dataset (R2R). The results show that the adversarial training process with our proposed APS benefits VLN models under both seen and unseen environments. And the pre-exploration process can further gain additional improvements under unseen environments.