# Sub-Instruction Aware Vision-and-Language Navigation

arXiv: https://arxiv.org/abs/2004.02707 · HTML: https://mlsanity.com/paper/2004.02707/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2020.

## Built on by (most cited first, 30 of 66)

- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [Towards Learning a Generalist Model for Embodied Navigation](https://mlsanity.com/paper/2312.02010/) (2023)
- [Airbert: In-domain Pretraining for Vision-and-Language Navigation](https://mlsanity.com/paper/2108.09105/) (2021)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [Structured Scene Memory for Vision-Language Navigation](https://mlsanity.com/paper/2103.03454/) (2021)
- [Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2109.15207/) (2021)
- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation](https://mlsanity.com/paper/2305.19195/) (2023)
- [Vision-Language Navigation with Random Environmental Mixup](https://mlsanity.com/paper/2106.07876/) (2021)
- [A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models](https://mlsanity.com/paper/2308.07997/) (2023)
- [Vision-Language Navigation: A Survey and Taxonomy](https://mlsanity.com/paper/2108.11544/) (2021)
- [March in Chat: Interactive Prompting for Remote Embodied Referring Expression](https://mlsanity.com/paper/2308.10141/) (2023)
- [CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memory](https://mlsanity.com/paper/2505.05622/) (2025)
- [Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information](https://mlsanity.com/paper/2104.09580/) (2021)
- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2106.13948/) (2021)
- [Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation](https://mlsanity.com/paper/2308.12587/) (2023)
- [One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones](https://mlsanity.com/paper/2202.07028/) (2022)
- [AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments](https://mlsanity.com/paper/2210.07940/) (2022)
- [VLN-Trans: Translator for the Vision and Language Navigation Agent](https://mlsanity.com/paper/2302.09230/) (2023)
- [NavBench: Probing Multimodal Large Language Models for Embodied Navigation](https://mlsanity.com/paper/2506.01031/) (2025)
- [Controllable Navigation Instruction Generation with Chain of Thought Prompting](https://mlsanity.com/paper/2407.07433/) (2024)
- [WebVLN: Vision-and-Language Navigation on Websites](https://mlsanity.com/paper/2312.15820/) (2023)
- [VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation](https://mlsanity.com/paper/2512.19021/) (2025)
- [Curriculum Learning for Vision-and-Language Navigation](https://mlsanity.com/paper/2111.07228/) (2021)
- [Correctable Landmark Discovery via Large Models for Vision-Language Navigation](https://mlsanity.com/paper/2405.18721/) (2024)
- [FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Tasks](https://mlsanity.com/paper/2503.13966/) (2025)
- [Contrastive Instruction-Trajectory Learning for Vision-Language Navigation](https://mlsanity.com/paper/2112.04138/) (2021)
- [Towards Navigation by Reasoning over Spatial Configurations](https://mlsanity.com/paper/2105.06839/) (2021)
- [Do Visual Imaginations Improve Vision-and-Language Navigation Agents?](https://mlsanity.com/paper/2503.16394/) (2025)
- [Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generation](https://mlsanity.com/paper/2307.13368/) (2023)

## Abstract (arXiv)

Vision-and-language navigation requires an agent to navigate through a real 3D environment following natural language instructions. Despite significant advances, few previous works are able to fully utilize the strong correspondence between the visual and textual sequences. Meanwhile, due to the lack of intermediate supervision, the agent's performance at following each part of the instruction cannot be assessed during navigation. In this work, we focus on the granularity of the visual and language sequences as well as the traceability of agents through the completion of an instruction. We provide agents with fine-grained annotations during training and find that they are able to follow the instruction better and have a higher chance of reaching the target at test time. We enrich the benchmark dataset Room-to-Room (R2R) with sub-instructions and their corresponding paths. To make use of this data, we propose effective sub-instruction attention and shifting modules that select and attend to a single sub-instruction at each time-step. We implement our sub-instruction modules in four state-of-the-art agents, compare with their baseline models, and show that our proposed method improves the performance of all four agents. We release the Fine-Grained R2R dataset (FGR2R) and the code at https://github.com/YicongHong/Fine-Grained-R2R.