# BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps

arXiv: https://arxiv.org/abs/2005.04625 · HTML: https://mlsanity.com/paper/2005.04625/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2020.

## Built on by (most cited first, 30 of 49)

- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [Episodic Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2105.06453/) (2021)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2109.15207/) (2021)
- [Cross-modal Map Learning for Vision and Language Navigation](https://mlsanity.com/paper/2203.05137/) (2022)
- [Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning](https://mlsanity.com/paper/2507.16746/) (2025)
- [EnvEdit: Environment Editing for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.15685/) (2022)
- [Vision-Language Navigation with Random Environmental Mixup](https://mlsanity.com/paper/2106.07876/) (2021)
- [VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View](https://mlsanity.com/paper/2307.06082/) (2023)
- [Target-Driven Structured Transformer Planner for Vision-Language Navigation](https://mlsanity.com/paper/2207.11201/) (2022)
- [ADAPT: Vision-Language Navigation with Modality-Aligned Action Prompts](https://mlsanity.com/paper/2205.15509/) (2022)
- [Word Acquisition in Neural Language Models](https://mlsanity.com/paper/2110.02406/) (2021)
- [Iterative Vision-and-Language Navigation](https://mlsanity.com/paper/2210.03087/) (2022)
- [Vision-Language Navigation: A Survey and Taxonomy](https://mlsanity.com/paper/2108.11544/) (2021)
- [Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information](https://mlsanity.com/paper/2104.09580/) (2021)
- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2106.13948/) (2021)
- [Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation](https://mlsanity.com/paper/2308.12587/) (2023)
- [One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones](https://mlsanity.com/paper/2202.07028/) (2022)
- [AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments](https://mlsanity.com/paper/2210.07940/) (2022)
- [FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigation](https://mlsanity.com/paper/2206.04294/) (2022)
- [Controllable Navigation Instruction Generation with Chain of Thought Prompting](https://mlsanity.com/paper/2407.07433/) (2024)
- [Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel](https://mlsanity.com/paper/2412.08467/) (2024)
- [WebVLN: Vision-and-Language Navigation on Websites](https://mlsanity.com/paper/2312.15820/) (2023)
- [VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation](https://mlsanity.com/paper/2512.19021/) (2025)
- [Curriculum Learning for Vision-and-Language Navigation](https://mlsanity.com/paper/2111.07228/) (2021)
- [FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Tasks](https://mlsanity.com/paper/2503.13966/) (2025)
- [Contrastive Instruction-Trajectory Learning for Vision-Language Navigation](https://mlsanity.com/paper/2112.04138/) (2021)
- [Towards Navigation by Reasoning over Spatial Configurations](https://mlsanity.com/paper/2105.06839/) (2021)
- [CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments](https://mlsanity.com/paper/2306.04047/) (2023)

## Abstract (arXiv)

Learning to follow instructions is of fundamental importance to autonomous agents for vision-and-language navigation (VLN). In this paper, we study how an agent can navigate long paths when learning from a corpus that consists of shorter ones. We show that existing state-of-the-art agents do not generalize well. To this end, we propose BabyWalk, a new VLN agent that is learned to navigate by decomposing long instructions into shorter ones (BabySteps) and completing them sequentially. A special design memory buffer is used by the agent to turn its past experiences into contexts for future steps. The learning process is composed of two phases. In the first phase, the agent uses imitation learning from demonstration to accomplish BabySteps. In the second phase, the agent uses curriculum-based reinforcement learning to maximize rewards on navigation tasks with increasingly longer instructions. We create two new benchmark datasets (of long navigation tasks) and use them in conjunction with existing ones to examine BabyWalk's generalization ability. Empirical results show that BabyWalk achieves state-of-the-art results on several metrics, in particular, is able to follow long instructions better. The codes and the datasets are released on our project page https://github.com/Sha-Lab/babywalk.