# Transferable Representation Learning in Vision-and-Language Navigation

arXiv: https://arxiv.org/abs/1908.03409 · HTML: https://mlsanity.com/paper/1908.03409/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Aug 2019.

## Built on by (most cited first, 30 of 48)

- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [Episodic Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2105.06453/) (2021)
- [Airbert: In-domain Pretraining for Vision-and-Language Navigation](https://mlsanity.com/paper/2108.09105/) (2021)
- [SOON: Scenario Oriented Object Navigation with Graph-based Exploration](https://mlsanity.com/paper/2103.17138/) (2021)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models](https://mlsanity.com/paper/2407.12366/) (2024)
- [Structured Scene Memory for Vision-Language Navigation](https://mlsanity.com/paper/2103.03454/) (2021)
- [DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation](https://mlsanity.com/paper/2308.07498/) (2023)
- [EnvEdit: Environment Editing for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.15685/) (2022)
- [Bird's-Eye-View Scene Graph for Vision-Language Navigation](https://mlsanity.com/paper/2308.04758/) (2023)
- [Vision-Language Navigation with Random Environmental Mixup](https://mlsanity.com/paper/2106.07876/) (2021)
- [HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.11591/) (2022)
- [Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMs](https://mlsanity.com/paper/2409.18794/) (2024)
- [Target-Driven Structured Transformer Planner for Vision-Language Navigation](https://mlsanity.com/paper/2207.11201/) (2022)
- [CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation](https://mlsanity.com/paper/2211.16649/) (2022)
- [Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigation](https://mlsanity.com/paper/2203.16586/) (2022)
- [Learning Vision-and-Language Navigation from YouTube Videos](https://mlsanity.com/paper/2307.11984/) (2023)
- [On the Evaluation of Vision-and-Language Navigation Instructions](https://mlsanity.com/paper/2101.10504/) (2021)
- [ADAPT: Vision-Language Navigation with Modality-Aligned Action Prompts](https://mlsanity.com/paper/2205.15509/) (2022)
- [Lana: A Language-Capable Navigator for Instruction Following and Generation](https://mlsanity.com/paper/2303.08409/) (2023)
- [Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision](https://mlsanity.com/paper/2504.02477/) (2025)
- [Diagnosing Vision-and-Language Navigation: What Really Matters](https://mlsanity.com/paper/2103.16561/) (2021)
- [Vision-Language Navigation: A Survey and Taxonomy](https://mlsanity.com/paper/2108.11544/) (2021)
- [Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information](https://mlsanity.com/paper/2104.09580/) (2021)
- [Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation](https://mlsanity.com/paper/2308.12587/) (2023)
- [Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration](https://mlsanity.com/paper/2203.04006/) (2022)
- [MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chains](https://mlsanity.com/paper/2405.10620/) (2024)
- [Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigation](https://mlsanity.com/paper/2309.08138/) (2023)
- [Correctable Landmark Discovery via Large Models for Vision-Language Navigation](https://mlsanity.com/paper/2405.18721/) (2024)
- [Mind the Error! Detection and Localization of Instruction Errors in Vision-and-Language Navigation](https://mlsanity.com/paper/2403.10700/) (2024)

## Abstract (arXiv)

Vision-and-Language Navigation (VLN) tasks such as Room-to-Room (R2R) require machine agents to interpret natural language instructions and learn to act in visually realistic environments to achieve navigation goals. The overall task requires competence in several perception problems: successful agents combine spatio-temporal, vision and language understanding to produce appropriate action sequences. Our approach adapts pre-trained vision and language representations to relevant in-domain tasks making them more effective for VLN. Specifically, the representations are adapted to solve both a cross-modal sequence alignment and sequence coherence task. In the sequence alignment task, the model determines whether an instruction corresponds to a sequence of visual frames. In the sequence coherence task, the model determines whether the perceptual sequences are predictive sequentially in the instruction-conditioned latent space. By transferring the domain-adapted representations, we improve competitive agents in R2R as measured by the success rate weighted by path length (SPL) metric.