# CityWalker: Learning Embodied Urban Navigation from Web-Scale Videos

arXiv: https://arxiv.org/abs/2411.17820 · HTML: https://mlsanity.com/paper/2411.17820/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 66 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 30 of 58)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2004.02857/) (2020)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration](https://mlsanity.com/paper/2310.07896/) (2023)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [ViNT: A Foundation Model for Visual Navigation](https://mlsanity.com/paper/2306.14846/) (2023)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [Semi-parametric Topological Memory for Navigation](https://mlsanity.com/paper/1803.00653/) (2018)
- [Deep Patch Visual Odometry](https://mlsanity.com/paper/2208.04726/) (2022)
- [ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings](https://mlsanity.com/paper/2206.12403/) (2022)
- [GNM: A General Navigation Model to Drive Any Robot](https://mlsanity.com/paper/2210.03370/) (2022)

## Built on by (most cited first, 30 of 41)

- [Astra: General Interactive World Model with Autoregressive Denoising](https://mlsanity.com/paper/2512.08931/) (2025)
- [ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigation](https://mlsanity.com/paper/2602.11598/) (2026)
- [Learning to Drive Anywhere with Model-Based Reannotation](https://mlsanity.com/paper/2505.05592/) (2025)
- [NaviTrace: Evaluating Embodied Navigation of Vision-Language Models](https://mlsanity.com/paper/2510.26909/) (2025)
- [From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning](https://mlsanity.com/paper/2507.22028/) (2025)
- [SocialNav: Training Human-Inspired Foundation Model for Socially-Aware Embodied Navigation](https://mlsanity.com/paper/2511.21135/) (2025)
- [Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigation](https://mlsanity.com/paper/2606.15869/) (2026)
- [CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?](https://mlsanity.com/paper/2512.16755/) (2025)
- [CapNav: Benchmarking Vision Language Models on Capability-conditioned Indoor Navigation](https://mlsanity.com/paper/2602.18424/) (2026)
- [ABot-N1: Toward a General Visual Language Navigation Foundation Model](https://mlsanity.com/paper/2607.10383/) (2026)
- [UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videos](https://mlsanity.com/paper/2510.15018/) (2025)
- [NavWM: A Unified Navigation World Model for Foresight-Driven Planning](https://mlsanity.com/paper/2606.24101/) (2026)
- [GrndCtrl: Grounding World Models via Self-Supervised Reward Alignment](https://mlsanity.com/paper/2512.01952/) (2025)
- [MM-Nav: Multi-View VLA Model for Robust Visual Navigation via Multi-Expert Learning](https://mlsanity.com/paper/2510.03142/) (2025)
- [UrbanVLA: A Vision-Language-Action Model for Urban Micromobility](https://mlsanity.com/paper/2510.23576/) (2025)
- [Wanderland: Geometrically Grounded Simulation for Open-World Embodied AI](https://mlsanity.com/paper/2511.20620/) (2025)
- [Bridging the Indoor-Outdoor Gap: Vision-Centric Instruction-Guided Embodied Navigation for the Last Meters](https://mlsanity.com/paper/2602.06427/) (2026)
- [Narrate2Nav: Real-Time Visual Navigation with Implicit Language Reasoning in Human-Centric Environments](https://mlsanity.com/paper/2506.14233/) (2025)
- [Target-Bench: Can Video World Models Achieve Mapless Path Planning with Semantic Targets?](https://mlsanity.com/paper/2511.17792/) (2025)
- [VLD: Visual Language Goal Distance for Reinforcement Learning Navigation](https://mlsanity.com/paper/2512.07976/) (2025)
- [STRNet: Visual Navigation with Spatio-Temporal Representation through Dynamic Graph Aggregation](https://mlsanity.com/paper/2604.02829/) (2026)
- [Visually-grounded Humanoid Agents](https://mlsanity.com/paper/2604.08509/) (2026)
- [POINav: Benchmarking and Enhancing Final-Meters Arrival in Real-World Vision-Language Navigation](https://mlsanity.com/paper/2605.28237/) (2026)
- [Exploring the Use of VLMs for Navigation Assistance for People with Blindness and Low Vision](https://mlsanity.com/paper/2603.15624/) (2026)
- [Learning Sidewalk Autopilot from Multi-Scale Imitation with Corrective Behavior Expansion](https://mlsanity.com/paper/2603.22527/) (2026)
- [360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents](https://mlsanity.com/paper/2608.08814/) (2026)
- [UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City](https://mlsanity.com/paper/2608.27456/) (2026)
- [YOPO-Nav: Visual Navigation using 3DGS Graphs from One-Pass Videos](https://mlsanity.com/paper/2512.09903/) (2025)
- [Stereo4DWalker: Learning 4D-aware Embodied Urban Navigation from Internet Stereo Videos](https://mlsanity.com/paper/2512.10956/) (2025)
- [RVN-Bench: A Benchmark for Reactive Visual Navigation](https://mlsanity.com/paper/2603.03953/) (2026)

## Abstract (arXiv)

Navigating dynamic urban environments presents significant challenges for embodied agents, requiring advanced spatial reasoning and adherence to common-sense norms. Despite progress, existing visual navigation methods struggle in map-free or off-street settings, limiting the deployment of autonomous agents like last-mile delivery robots. To overcome these obstacles, we propose a scalable, data-driven approach for human-like urban navigation by training agents on thousands of hours of in-the-wild city walking and driving videos sourced from the web. We introduce a simple and scalable data processing pipeline that extracts action supervision from these videos, enabling large-scale imitation learning without costly annotations. Our model learns sophisticated navigation policies to handle diverse challenges and critical scenarios. Experimental results show that training on large-scale, diverse datasets significantly enhances navigation performance, surpassing current methods. This work shows the potential of using abundant online video data to develop robust navigation policies for embodied agents in dynamic urban settings. Project homepage is at https://ai4ce.github.io/CityWalker/.