# Environment-agnostic Multitask Learning for Natural Language Grounded Navigation

arXiv: https://arxiv.org/abs/2003.00443 · HTML: https://mlsanity.com/paper/2003.00443/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2020.

## Built on by (most cited first, 30 of 43)

- [History Aware Multimodal Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2110.13309/) (2021)
- [Towards Learning a Generalist Model for Embodied Navigation](https://mlsanity.com/paper/2312.02010/) (2023)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [Scaling Data Generation in Vision-and-Language Navigation](https://mlsanity.com/paper/2307.15644/) (2023)
- [Structured Scene Memory for Vision-Language Navigation](https://mlsanity.com/paper/2103.03454/) (2021)
- [EnvEdit: Environment Editing for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.15685/) (2022)
- [PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation](https://mlsanity.com/paper/2305.19195/) (2023)
- [Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation](https://mlsanity.com/paper/2202.02440/) (2022)
- [Target-Driven Structured Transformer Planner for Vision-Language Navigation](https://mlsanity.com/paper/2207.11201/) (2022)
- [Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigation](https://mlsanity.com/paper/2203.16586/) (2022)
- [Neighbor-view Enhanced Model for Vision and Language Navigation](https://mlsanity.com/paper/2107.07201/) (2021)
- [Improving Vision-and-Language Navigation by Generating Future-View Image Semantics](https://mlsanity.com/paper/2304.04907/) (2023)
- [Diagnosing Vision-and-Language Navigation: What Really Matters](https://mlsanity.com/paper/2103.16561/) (2021)
- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2106.13948/) (2021)
- [Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation](https://mlsanity.com/paper/2308.12587/) (2023)
- [Towards Versatile Embodied Navigation](https://mlsanity.com/paper/2210.16822/) (2022)
- [RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigation](https://mlsanity.com/paper/2412.08591/) (2024)
- [Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding](https://mlsanity.com/paper/2207.08386/) (2022)
- [Curriculum Learning for Vision-and-Language Navigation](https://mlsanity.com/paper/2111.07228/) (2021)
- [SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts](https://mlsanity.com/paper/2412.05552/) (2024)
- [Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation](https://mlsanity.com/paper/2107.11252/) (2021)
- [VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator](https://mlsanity.com/paper/2105.11589/) (2021)
- [CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations](https://mlsanity.com/paper/2207.02185/) (2022)
- [ESceme: Vision-and-Language Navigation with Episodic Scene Memory](https://mlsanity.com/paper/2303.01032/) (2023)
- [Structure-Encoding Auxiliary Tasks for Improved Visual Representation in Vision-and-Language Navigation](https://mlsanity.com/paper/2211.11116/) (2022)
- [DELAN: Dual-Level Alignment for Vision-and-Language Navigation by Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2404.01994/) (2024)
- [Rethinking the Spatial Route Prior in Vision-and-Language Navigation](https://mlsanity.com/paper/2110.05728/) (2021)
- [Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning](https://mlsanity.com/paper/2403.05770/) (2024)
- [NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigation](https://mlsanity.com/paper/2510.16457/) (2025)
- [Trajectory-Diversity-Driven Robust Vision-and-Language Navigation](https://mlsanity.com/paper/2603.15370/) (2026)

## Abstract (arXiv)

Recent research efforts enable study for natural language grounded navigation in photo-realistic environments, e.g., following natural language instructions or dialog. However, existing methods tend to overfit training data in seen environments and fail to generalize well in previously unseen environments. To close the gap between seen and unseen environments, we aim at learning a generalized navigation model from two novel perspectives: (1) we introduce a multitask navigation model that can be seamlessly trained on both Vision-Language Navigation (VLN) and Navigation from Dialog History (NDH) tasks, which benefits from richer natural language guidance and effectively transfers knowledge across tasks; (2) we propose to learn environment-agnostic representations for the navigation policy that are invariant among the environments seen during training, thus generalizing better on unseen environments. Extensive experiments show that environment-agnostic multitask learning significantly reduces the performance gap between seen and unseen environments, and the navigation agent trained so outperforms baselines on unseen environments by 16% (relative measure on success rate) on VLN and 120% (goal progress) on NDH. Our submission to the CVDN leaderboard establishes a new state-of-the-art for the NDH task on the holdout test set. Code is available at https://github.com/google-research/valan.