# EnvEdit: Environment Editing for Vision-and-Language Navigation

arXiv: https://arxiv.org/abs/2203.15685 · HTML: https://mlsanity.com/paper/2203.15685/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 81 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 30 of 68)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [A Learned Representation For Artistic Style](https://mlsanity.com/paper/1610.07629/) (2016)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [A Recurrent Vision-and-Language BERT for Navigation](https://mlsanity.com/paper/2011.13922/) (2020)
- [History Aware Multimodal Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2110.13309/) (2021)
- [Vision-and-Dialog Navigation](https://mlsanity.com/paper/1907.04957/) (2019)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Geometric GAN](https://mlsanity.com/paper/1705.02894/) (2017)
- [Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training](https://mlsanity.com/paper/2002.10638/) (2020)
- [Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks](https://mlsanity.com/paper/1911.07883/) (2019)

## Built on by (most cited first, 30 of 83)

- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2304.03047/) (2023)
- [GridMM: Grid Memory Map for Vision-and-Language Navigation](https://mlsanity.com/paper/2307.12907/) (2023)
- [BEVBert: Multimodal Map Pre-training for Language-guided Navigation](https://mlsanity.com/paper/2212.04385/) (2022)
- [Scaling Data Generation in Vision-and-Language Navigation](https://mlsanity.com/paper/2307.15644/) (2023)
- [Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions](https://mlsanity.com/paper/2309.11382/) (2023)
- [NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models](https://mlsanity.com/paper/2407.12366/) (2024)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation](https://mlsanity.com/paper/2305.19195/) (2023)
- [Bird's-Eye-View Scene Graph for Vision-Language Navigation](https://mlsanity.com/paper/2308.04758/) (2023)
- [OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNav](https://mlsanity.com/paper/2303.07798/) (2023)
- [VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View](https://mlsanity.com/paper/2307.06082/) (2023)
- [Volumetric Environment Representation for Vision-Language Navigation](https://mlsanity.com/paper/2403.14158/) (2024)
- [Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMs](https://mlsanity.com/paper/2409.18794/) (2024)
- [A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning](https://mlsanity.com/paper/2210.03112/) (2022)
- [CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation](https://mlsanity.com/paper/2211.16649/) (2022)
- [Vision-and-Language Navigation via Causal Learning](https://mlsanity.com/paper/2404.10241/) (2024)
- [Learning from Unlabeled 3D Environments for Vision-and-Language Navigation](https://mlsanity.com/paper/2208.11781/) (2022)
- [Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2412.10137/) (2024)
- [Learning Vision-and-Language Navigation from YouTube Videos](https://mlsanity.com/paper/2307.11984/) (2023)
- [Improving Vision-and-Language Navigation by Generating Future-View Image Semantics](https://mlsanity.com/paper/2304.04907/) (2023)
- [EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents](https://mlsanity.com/paper/2403.12014/) (2024)
- [Learning Navigational Visual Representations with Semantic Map Supervision](https://mlsanity.com/paper/2307.12335/) (2023)
- [Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision](https://mlsanity.com/paper/2504.02477/) (2025)
- [1st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022)](https://mlsanity.com/paper/2206.11610/) (2022)
- [Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation](https://mlsanity.com/paper/2308.12587/) (2023)
- [Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigation](https://mlsanity.com/paper/2311.13209/) (2023)
- [DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback](https://mlsanity.com/paper/2410.06215/) (2024)
- [VLN-PETL: Parameter-Efficient Transfer Learning for Vision-and-Language Navigation](https://mlsanity.com/paper/2308.10172/) (2023)

## Abstract (arXiv)

In Vision-and-Language Navigation (VLN), an agent needs to navigate through the environment based on natural language instructions. Due to limited available data for agent training and finite diversity in navigation environments, it is challenging for the agent to generalize to new, unseen environments. To address this problem, we propose EnvEdit, a data augmentation method that creates new environments by editing existing environments, which are used to train a more generalizable agent. Our augmented environments can differ from the seen environments in three diverse aspects: style, object appearance, and object classes. Training on these edit-augmented environments prevents the agent from overfitting to existing environments and helps generalize better to new, unseen environments. Empirically, on both the Room-to-Room and the multi-lingual Room-Across-Room datasets, we show that our proposed EnvEdit method gets significant improvements in all metrics on both pre-trained and non-pre-trained VLN agents, and achieves the new state-of-the-art on the test leaderboard. We further ensemble the VLN agents augmented on different edited environments and show that these edit methods are complementary. Code and data are available at https://github.com/jialuli-luka/EnvEdit