# Learning To Follow Directions in Street View

arXiv: https://arxiv.org/abs/1903.00401 · HTML: https://mlsanity.com/paper/1903.00401/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2019.

## Built on by (most cited first, 29 of 29)

- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Waypoint Models for Instruction-guided Navigation in Continuous Environments](https://mlsanity.com/paper/2110.02207/) (2021)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [EnvEdit: Environment Editing for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.15685/) (2022)
- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View](https://mlsanity.com/paper/2307.06082/) (2023)
- [CityNav: A Large-Scale Dataset for Real-World Aerial Navigation](https://mlsanity.com/paper/2406.14240/) (2024)
- [CityWalker: Learning Embodied Urban Navigation from Web-Scale Videos](https://mlsanity.com/paper/2411.17820/) (2024)
- [Vision-Language Navigation: A Survey and Taxonomy](https://mlsanity.com/paper/2108.11544/) (2021)
- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2106.13948/) (2021)
- [Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation](https://mlsanity.com/paper/2308.12587/) (2023)
- [Look Wide and Interpret Twice: Improving Performance on Interactive Instruction-following Tasks](https://mlsanity.com/paper/2106.00596/) (2021)
- [Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions](https://mlsanity.com/paper/2408.04168/) (2024)
- [Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas](https://mlsanity.com/paper/2203.13838/) (2022)
- [Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Grounding](https://mlsanity.com/paper/2303.04077/) (2023)
- [Learning Local Implicit Fourier Representation for Image Warping](https://mlsanity.com/paper/2207.01831/) (2022)
- [VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigation](https://mlsanity.com/paper/2402.03561/) (2024)
- [Structure-Encoding Auxiliary Tasks for Improved Visual Representation in Vision-and-Language Navigation](https://mlsanity.com/paper/2211.11116/) (2022)
- [Vision-Language Navigation with Embodied Intelligence: A Survey](https://mlsanity.com/paper/2402.14304/) (2024)
- [DOROTHIE: Spoken Dialogue for Handling Unexpected Situations in Interactive Autonomous Driving Agents](https://mlsanity.com/paper/2210.12511/) (2022)
- [Semantic Map-based Generation of Navigation Instructions](https://mlsanity.com/paper/2403.19603/) (2024)
- [A Priority Map for Vision-and-Language Navigation with Trajectory Plans and Feature-Location Cues](https://mlsanity.com/paper/2207.11717/) (2022)
- [ViLPAct: A Benchmark for Compositional Generalization on Multimodal Human Activities](https://mlsanity.com/paper/2210.05556/) (2022)
- [Can ChatGPT assist visually impaired people with micro-navigation?](https://mlsanity.com/paper/2408.08321/) (2024)
- [City Navigation in the Wild: Exploring Emergent Navigation from Web-Scale Knowledge in MLLMs](https://mlsanity.com/paper/2512.15933/) (2025)
- [A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigation](https://mlsanity.com/paper/2607.09792/) (2026)
- [Are you doing what I say? On modalities alignment in ALFRED](https://mlsanity.com/paper/2110.05665/) (2021)
- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2304.02738/) (2023)
- [Multi-model fusion for Aerial Vision and Dialog Navigation based on human attention aids](https://mlsanity.com/paper/2308.14064/) (2023)

## Abstract (arXiv)

Navigating and understanding the real world remains a key challenge in machine learning and inspires a great variety of research in areas such as language grounding, planning, navigation and computer vision. We propose an instruction-following task that requires all of the above, and which combines the practicality of simulated environments with the challenges of ambiguous, noisy real world data. StreetNav is built on top of Google Street View and provides visually accurate environments representing real places. Agents are given driving instructions which they must learn to interpret in order to successfully navigate in this environment. Since humans equipped with driving instructions can readily navigate in previously unseen cities, we set a high bar and test our trained agents for similar cognitive capabilities. Although deep reinforcement learning (RL) methods are frequently evaluated only on data that closely follow the training distribution, our dataset extends to multiple cities and has a clean train/test separation. This allows for thorough testing of generalisation ability. This paper presents the StreetNav environment and tasks, models that establish strong baselines, and extensive analysis of the task and the trained agents.