mlsanityData to 14 Oct 2026

Cognitive Mapping and Planning for Visual Navigation

arXiv 1702.03920 ↗First posted 13 Feb 2017cs.CV cs.AI cs.LGcited by 212

Not on the map. Why?

Abstract (arXiv)

We introduce a neural architecture for navigation in novel environments. Our proposed architecture learns to map from first-person views and plans a sequence of actions towards goals in the environment. The Cognitive Mapper and Planner (CMP) is based on two key ideas: a) a unified joint architecture for mapping and planning, such that the mapping is driven by the needs of the task, and b) a spatial memory with the ability to plan given an incomplete set of observations about the world. CMP constructs a top-down belief map of the world and applies a differentiable neural net planner to produce the next action at each time step. The accumulated belief of the world enables the agent to track visited regions of the environment. We train and test CMP on navigation problems in simulation environments derived from scans of real world buildings. Our experiments demonstrate that CMP outperforms alternate learning-based architectures, as well as, classical mapping and path planning approaches in many cases. Furthermore, it naturally extends to semantically specified goals, such as 'going to a chair'. We also deploy CMP on physical robots in indoor environments, where it achieves reasonable performance, even though it is trained entirely in simulation.

Key facts. Cited by 212 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 2; 2020: 1; 2021: 70; 2022: 57; 2023: 25; 2024: 21; 2025: 17; 2026: 18.

212 papers.

Sort:

By area:

  1. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  2. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  3. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 261
  4. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  5. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  6. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  7. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 150
  8. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 132
  9. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 124
  10. 2019Imitation Learning as f-Divergence Minimizationcited by 123
  11. 2021FILM: Following Instructions in Language with Modular Methodscited by 121
  12. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 119
38 more
  1. 2021Capturing, Reconstructing, and Simulating: the UrbanScene3D Datasetcited by 118
  2. 2022Navigating to Objects in the Real Worldcited by 106
  3. 2021Visual Room Rearrangementcited by 102
  4. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 102
  5. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  6. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  7. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  8. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 79
  9. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  10. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 67
  11. 2021Memory-Augmented Reinforcement Learning for Image-Goal Navigationcited by 66
  12. 2022A Survey on Programmatic Weak Supervisioncited by 66
  13. 2021The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AIcited by 65
  14. 2021No RL, No Simulation: Learning to Navigate without Navigatingcited by 65
  15. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  16. 2021RobustNav: Towards Benchmarking Robustness in Embodied Navigationcited by 54
  17. 2021Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networkscited by 52
  18. 2021Coupling Vision and Proprioception for Navigation of Legged Robotscited by 51
  19. 2023Renderable Neural Radiance Map for Visual Navigationcited by 51
  20. 2021Learning to Map for Active Semantic Goal Navigationcited by 50
  21. 2021Scene-Intuitive Agent for Remote Embodied Visual Groundingcited by 49
  22. 2021SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environmentscited by 48
  23. 2022Sound Adversarial Audio-Visual Navigationcited by 48
  24. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 46
  25. 2022Retrospectives on the Embodied AI Workshopcited by 46
  26. 20223D-Aware Object Goal Navigation via Simultaneous Exploration and Identificationcited by 46
  27. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  28. 2021Visual Navigation with Spatial Attentioncited by 43
  29. 2021Catch Me If You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments with Moving Soundscited by 41
  30. 2024CityWalker: Learning Embodied Urban Navigation from Web-Scale Videoscited by 41
  31. 2022Continuous Scene Representations for Embodied AIcited by 39
  32. 2022DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehiclescited by 37
  33. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  34. 2021Differentiable SLAM-net: Learning Particle SLAM for Visual Navigationcited by 34
  35. 2021Hierarchical Object-to-Zone Graph for Object Navigationcited by 34
  36. 2022Uncertainty-driven Planner for Exploration and Navigationcited by 33
  37. 2022Stubborn: A Strong Baseline for Indoor Object Navigationcited by 32
  38. 2022WayFAST: Navigation with Predictive Traversability in the Fieldcited by 32

50 most cited of 212 All 212 (JSON)