mlsanityData to 7 Oct 2026

CityWalker: Learning Embodied Urban Navigation from Web-Scale Videos

arXiv 2411.17820 ↗First posted 26 Nov 2024cs.CV cs.ROcited by 40

Not on the map. Why?

Abstract (arXiv)

Navigating dynamic urban environments presents significant challenges for embodied agents, requiring advanced spatial reasoning and adherence to common-sense norms. Despite progress, existing visual navigation methods struggle in map-free or off-street settings, limiting the deployment of autonomous agents like last-mile delivery robots. To overcome these obstacles, we propose a scalable, data-driven approach for human-like urban navigation by training agents on thousands of hours of in-the-wild city walking and driving videos sourced from the web. We introduce a simple and scalable data processing pipeline that extracts action supervision from these videos, enabling large-scale imitation learning without costly annotations. Our model learns sophisticated navigation policies to handle diverse challenges and critical scenarios. Experimental results show that training on large-scale, diverse datasets significantly enhances navigation performance, surpassing current methods. This work shows the potential of using abundant online video data to develop robust navigation policies for embodied agents in dynamic urban settings. Project homepage is at https://ai4ce.github.io/CityWalker/.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 58 of its 66 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

58 of 66 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023Segment Anythingcited by 8,158
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2017Mask R-CNNcited by 7,259
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  11. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 983
  12. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
38 more
  1. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  2. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 673
  3. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  4. 2016Direct Sparse Odometrycited by 479
  5. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  6. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  7. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  8. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  9. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  10. 2017Embodied Question Answeringcited by 335
  11. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  12. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 234
  13. 2023ViNT: A Foundation Model for Visual Navigationcited by 210
  14. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  15. 2018Semi-parametric Topological Memory for Navigationcited by 203
  16. 2022Deep Patch Visual Odometrycited by 186
  17. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  18. 2022GNM: A General Navigation Model to Drive Any Robotcited by 161
  19. 2022Human-to-Robot Imitation in the Wildcited by 157
  20. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  21. 2022Navigating to Objects in the Real Worldcited by 104
  22. 2018Learning to Navigate in Cities Without a Mapcited by 103
  23. 2020BADGR: An Autonomous Self-Supervised Learning-Based Navigation Systemcited by 97
  24. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  25. 2020FAST-LIO: A Fast, Robust LiDAR-inertial Odometry Package by Tightly-Coupled Iterated Kalman Filtercited by 70
  26. 2021No RL, No Simulation: Learning to Navigate without Navigatingcited by 63
  27. 2023SACSoN: Scalable Autonomous Control for Social Navigationcited by 60
  28. 2022ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hintscited by 55
  29. 2023Navigating to Objects Specified by Imagescited by 50
  30. 2024CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environmentscited by 43
  31. 2019The StreetLearn Environment and Datasetcited by 42
  32. 2022Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretrainingcited by 37
  33. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Modelcited by 36
  34. 2022Learning to Imitate Object Interactions from Internet Videoscited by 32
  35. 2024LeLaN: Learning A Language-Conditioned Navigation Policy from In-the-Wild Videoscited by 31
  36. 2019Learning To Follow Directions in Street Viewcited by 29
  37. 2022Rethinking Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in Navigationcited by 28
  38. 2020LaND: Learning to Navigate from Disengagementscited by 24

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 24.

40 papers.

Sort:

By area:

  1. 2025Astra: General Interactive World Model with Autoregressive Denoisingcited by 30
  2. 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 23
  3. 2025NaviTrace: Evaluating Embodied Navigation of Vision-Language Modelscited by 13
  4. 2025Learning to Drive Anywhere with Model-Based Reannotationcited by 12
  5. 2025SocialNav: Training Human-Inspired Foundation Model for Socially-Aware Embodied Navigationcited by 12
  6. 2025From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learningcited by 11
  7. 2026Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigationcited by 10
  8. 2025CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?cited by 8
  9. 2026CapNav: Benchmarking Vision Language Models on Capability-conditioned Indoor Navigationcited by 6
  10. 2025UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videoscited by 5
  11. 2026ABot-N1: Toward a General Visual Language Navigation Foundation Modelcited by 5
  12. 2025GrndCtrl: Grounding World Models via Self-Supervised Reward Alignmentcited by 4
28 more
  1. 2026NavWM: A Unified Navigation World Model for Foresight-Driven Planningcited by 4
  2. 2025MM-Nav: Multi-View VLA Model for Robust Visual Navigation via Multi-Expert Learningcited by 3
  3. 2025Wanderland: Geometrically Grounded Simulation for Open-World Embodied AIcited by 3
  4. 2026Bridging the Indoor-Outdoor Gap: Vision-Centric Instruction-Guided Embodied Navigation for the Last Meterscited by 3
  5. 2025Narrate2Nav: Real-Time Visual Navigation with Implicit Language Reasoning in Human-Centric Environmentscited by 2
  6. 2025UrbanVLA: A Vision-Language-Action Model for Urban Micromobilitycited by 2
  7. 2025Target-Bench: Can Video World Models Achieve Mapless Path Planning with Semantic Targets?cited by 2
  8. 2025VLD: Visual Language Goal Distance for Reinforcement Learning Navigationcited by 2
  9. 2026STRNet: Visual Navigation with Spatio-Temporal Representation through Dynamic Graph Aggregationcited by 2
  10. 2026POINav: Benchmarking and Enhancing Final-Meters Arrival in Real-World Vision-Language Navigationcited by 2
  11. 2026Exploring the Use of VLMs for Navigation Assistance for People with Blindness and Low Visioncited by 1
  12. 2026Learning Sidewalk Autopilot from Multi-Scale Imitation with Corrective Behavior Expansioncited by 1
  13. 2026Visually-grounded Humanoid Agentscited by 1
  14. 2026360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agentscited by 1
  15. 2026UrbanGround: From Local Perception to Spatial Agency in a Real-Scale Citycited by 1
  16. 2025YOPO-Nav: Visual Navigation using 3DGS Graphs from One-Pass Videoscited by 0
  17. 2025Stereo4DWalker: Learning 4D-aware Embodied Urban Navigation from Internet Stereo Videoscited by 0
  18. 2026RVN-Bench: A Benchmark for Reactive Visual Navigationcited by 0
  19. 2026AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Modelscited by 0
  20. 2026TARIC: Memory-Augmented Traversability-Aware Outdoor VLN under Interrupted Semantic Cuescited by 0
  21. 2026RescueBench: Can Embodied Agents Save Lives in the Wild ?cited by 0
  22. 2026From Imitation to Alignment: Human-Preference Flow Policies for Long-Horizon Sidewalk Navigationcited by 0
  23. 2026WildCity: A Real-World City-Scale Testbed for Rendering, Simulation, and Spatial Intelligencecited by 0
  24. 2026GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Modelcited by 0
  25. 2026HumAIN: Human-Aware Implicit Social Robot Navigationcited by 0
  26. 2026ACME: A Multi-Cultural, Multi-Embodiment Social-Navigation Datasetcited by 0
  27. 2026AdaVDR: Adaptive Tool Use and Reflection for Video Deep Researchcited by 0
  28. 2026Learning Social Navigation from Internet Videos in the Policy State Spacecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025UrbanNav: Learning Language-Guided Urban Navigation from Web-Scale Human Trajectories
  2. 2026Exposing the Long-tail in Embodied Urban Navigation via Scalable Learning from In-the-Wild Videos ↗
  3. 2024EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment
  4. 2025City Navigation in the Wild: Exploring Emergent Navigation from Web-Scale Knowledge in MLLMs
  5. 2022Model-Based Imitation Learning for Urban Driving
  6. 2025UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces
  7. 2026Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos
  8. 2019Urban Driving with Conditional Imitation Learning