# Vision-Language Navigation: A Survey and Taxonomy

arXiv: https://arxiv.org/abs/2108.11544 · HTML: https://mlsanity.com/paper/2108.11544/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 128 of its 174 references that are in the map.
- First posted on arXiv 26 Aug 2021.

## Builds on (most cited first, 30 of 128)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)

## Built on by (most cited first, 30 of 34)

- [NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2506.23468/) (2025)
- [VLN-Trans: Translator for the Vision and Language Navigation Agent](https://mlsanity.com/paper/2302.09230/) (2023)
- [Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions](https://mlsanity.com/paper/2408.04168/) (2024)
- [Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigation](https://mlsanity.com/paper/2509.16924/) (2025)
- [VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation](https://mlsanity.com/paper/2403.12415/) (2024)
- [Advancing Audio-Visual Navigation Through Multi-Agent Collaboration in 3D Environments](https://mlsanity.com/paper/2509.22698/) (2025)
- [Graph Neural Networks in Vision-Language Image Understanding: A Survey](https://mlsanity.com/paper/2303.03761/) (2023)
- [Aerial Vision-Language Navigation with a Unified Framework for Spatial, Temporal and Embodied Reasoning](https://mlsanity.com/paper/2512.08639/) (2025)
- [CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?](https://mlsanity.com/paper/2512.16755/) (2025)
- [HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactions](https://mlsanity.com/paper/2503.14229/) (2025)
- [Active Test-time Vision-Language Navigation](https://mlsanity.com/paper/2506.06630/) (2025)
- [ViLAM: Distilling Vision-Language Reasoning into Attention Maps for Social Robot Navigation](https://mlsanity.com/paper/2503.09820/) (2025)
- [APEX: A Decoupled Memory-based Explorer for Asynchronous Aerial Object Goal Navigation](https://mlsanity.com/paper/2602.00551/) (2026)
- [MapDream: Task-Driven Map Learning for Vision-Language Navigation](https://mlsanity.com/paper/2602.00222/) (2026)
- [Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigation](https://mlsanity.com/paper/2604.18223/) (2026)
- [CANVAS: Commonsense-Aware Navigation System for Intuitive Human-Robot Interaction](https://mlsanity.com/paper/2410.01273/) (2024)
- [Structured Observation Language for Efficient and Generalizable Vision-Language Navigation](https://mlsanity.com/paper/2603.27577/) (2026)
- [NaVIP: An Image-Centric Indoor Navigation Solution for Visually Impaired People](https://mlsanity.com/paper/2410.18109/) (2024)
- [To Ask or Not to Ask? Detecting Absence of Information in Vision and Language Navigation](https://mlsanity.com/paper/2411.05831/) (2024)
- [Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering](https://mlsanity.com/paper/2510.01483/) (2025)
- [DIV-Nav: Open-Vocabulary Spatial Relationships for Multi-Object Navigation](https://mlsanity.com/paper/2510.16518/) (2025)
- [The Essence of Balance for Self-Improving Agents in Vision-and-Language Navigation](https://mlsanity.com/paper/2604.19064/) (2026)
- [VLN-Pilot: Large Vision-Language Model as an Autonomous Indoor Drone Operator](https://mlsanity.com/paper/2602.05552/) (2026)
- [Generalizable Audio-Visual Navigation via Binaural Difference Attention and Action Transition Prediction](https://mlsanity.com/paper/2604.05007/) (2026)
- [How Far Are Large Multimodal Models from Human-Level Spatial Action? A Benchmark for Goal-Oriented Embodied Navigation in Urban Airspace](https://mlsanity.com/paper/2604.07973/) (2026)
- [TARIC: Memory-Augmented Traversability-Aware Outdoor VLN under Interrupted Semantic Cues](https://mlsanity.com/paper/2605.31121/) (2026)
- [Path-level Hindsight Instructions for Semantic Exploration in Vision-Language Navigation](https://mlsanity.com/paper/2607.01754/) (2026)
- [Seek to Segment: Active Perception for Panoramic Referring Segmentation](https://mlsanity.com/paper/2607.02497/) (2026)
- [Holo-Captioning: Toward the Text Equivalent of 3D Scenes](https://mlsanity.com/paper/2607.02908/) (2026)
- [Dynamic Context Adapters: Efficiently Infusing History into Vision-and-Language Models](https://mlsanity.com/paper/2608.10525/) (2026)

## Abstract (arXiv)

Vision-Language Navigation (VLN) tasks require an agent to follow human language instructions to navigate in previously unseen environments. This challenging field involving problems in natural language processing, computer vision, robotics, etc., has spawn many excellent works focusing on various VLN tasks. This paper provides a comprehensive survey and an insightful taxonomy of these tasks based on the different characteristics of language instructions in these tasks. Depending on whether the navigation instructions are given for once or multiple times, this paper divides the tasks into two categories, i.e., single-turn and multi-turn tasks. For single-turn tasks, we further subdivide them into goal-oriented and route-oriented based on whether the instructions designate a single goal location or specify a sequence of multiple locations. For multi-turn tasks, we subdivide them into passive and interactive tasks based on whether the agent is allowed to question the instruction or not. These tasks require different capabilities of the agent and entail various model designs. We identify progress made on the tasks and look into the limitations of existing VLN models and task settings. Finally, we discuss several open issues of VLN and point out some opportunities in the future, i.e., incorporating knowledge with VLN models and implementing them in the real physical world.