# The StreetLearn Environment and Dataset

arXiv: https://arxiv.org/abs/1903.01292 · HTML: https://mlsanity.com/paper/1903.01292/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2019.

## Built on by (most cited first, 30 of 42)

- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Pathdreamer: A World Model for Indoor Navigation](https://mlsanity.com/paper/2105.08756/) (2021)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [Extreme Rotation Estimation using Dense Correlation Volumes](https://mlsanity.com/paper/2104.13530/) (2021)
- [CityWalker: Learning Embodied Urban Navigation from Web-Scale Videos](https://mlsanity.com/paper/2411.17820/) (2024)
- [Towards mental time travel: a hierarchical memory for reinforcement learning agents](https://mlsanity.com/paper/2105.14039/) (2021)
- [The 8-Point Algorithm as an Inductive Bias for Relative Pose Prediction by ViTs](https://mlsanity.com/paper/2208.08988/) (2022)
- [OpenStreetView-5M: The Many Roads to Global Visual Geolocation](https://mlsanity.com/paper/2404.18873/) (2024)
- [Vision-Language Navigation: A Survey and Taxonomy](https://mlsanity.com/paper/2108.11544/) (2021)
- [MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors](https://mlsanity.com/paper/2409.15273/) (2024)
- [Geolocation with Real Human Gameplay Data: A Large-Scale Dataset and Human-Like Reasoning Framework](https://mlsanity.com/paper/2502.13759/) (2025)
- [Deep Learning for Camera Calibration and Beyond: A Survey](https://mlsanity.com/paper/2303.10559/) (2023)
- [Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Grounding](https://mlsanity.com/paper/2303.04077/) (2023)
- [NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigation](https://mlsanity.com/paper/2411.08579/) (2024)
- [Image-Based Geolocation Using Large Vision-Language Models](https://mlsanity.com/paper/2408.09474/) (2024)
- [A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services](https://mlsanity.com/paper/2304.14271/) (2023)
- [VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigation](https://mlsanity.com/paper/2402.03561/) (2024)
- [NYU-VPR: Long-Term Visual Place Recognition Benchmark with View Direction and Data Anonymization Influences](https://mlsanity.com/paper/2110.09004/) (2021)
- [FAR: Flexible, Accurate and Robust 6DoF Relative Camera Pose Estimation](https://mlsanity.com/paper/2403.03221/) (2024)
- [PANORAMA: The Rise of Omnidirectional Vision in the Embodied AI Era](https://mlsanity.com/paper/2509.12989/) (2025)
- [SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark](https://mlsanity.com/paper/2110.10661/) (2021)
- [Vision-Language Navigation with Embodied Intelligence: A Survey](https://mlsanity.com/paper/2402.14304/) (2024)
- [Graph schemas as abstractions for transfer learning, inference, and planning](https://mlsanity.com/paper/2302.07350/) (2023)
- [Rethinking Generic Camera Models for Deep Single Image Camera Calibration to Recover Rotation and Fisheye Distortion](https://mlsanity.com/paper/2111.12927/) (2021)
- [Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering](https://mlsanity.com/paper/2310.00362/) (2023)
- [Extreme Rotation Estimation in the Wild](https://mlsanity.com/paper/2411.07096/) (2024)
- [CapNav: Benchmarking Vision Language Models on Capability-conditioned Indoor Navigation](https://mlsanity.com/paper/2602.18424/) (2026)
- [Exploring and Learning Structure: Active Inference Approach in Navigational Agents](https://mlsanity.com/paper/2408.05982/) (2024)
- [Image-based Geolocalization by Ground-to-2.5D Map Matching](https://mlsanity.com/paper/2308.05993/) (2023)
- [PanGEA: The Panoramic Graph Environment Annotation Toolkit](https://mlsanity.com/paper/2103.12703/) (2021)

## Abstract (arXiv)

Navigation is a rich and well-grounded problem domain that drives progress in many different areas of research: perception, planning, memory, exploration, and optimisation in particular. Historically these challenges have been separately considered and solutions built that rely on stationary datasets - for example, recorded trajectories through an environment. These datasets cannot be used for decision-making and reinforcement learning, however, and in general the perspective of navigation as an interactive learning task, where the actions and behaviours of a learning agent are learned simultaneously with the perception and planning, is relatively unsupported. Thus, existing navigation benchmarks generally rely on static datasets (Geiger et al., 2013; Kendall et al., 2015) or simulators (Beattie et al., 2016; Shah et al., 2018). To support and validate research in end-to-end navigation, we present StreetLearn: an interactive, first-person, partially-observed visual environment that uses Google Street View for its photographic content and broad coverage, and give performance baselines for a challenging goal-driven navigation task. The environment code, baseline agent code, and the dataset are available at http://streetlearn.cc