mlsanityData to 14 Oct 2026

Visual Language Maps for Robot Navigation

arXiv 2210.05714 ↗First posted 11 Oct 2022cs.RO cs.AI cs.CLcited by 365

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Grounding language to the visual observations of a navigating agent can be performed using off-the-shelf visual-language models pretrained on Internet-scale data (e.g., image captions). While this is useful for matching images to natural language descriptions of object goals, it remains disjoint from the process of mapping the environment, so that it lacks the spatial precision of classic geometric maps. To address this problem, we propose VLMaps, a spatial map representation that directly fuses pretrained visual-language features with a 3D reconstruction of the physical world. VLMaps can be autonomously built from video feed on robots using standard exploration approaches and enables natural language indexing of the map without additional labeled data. Specifically, when combined with large language models (LLMs), VLMaps can be used to (i) translate natural language commands into a sequence of open-vocabulary navigation goals (which, beyond prior work, can be spatial by construction, e.g., "in between the sofa and TV" or "three meters to the right of the chair") directly localized in the map, and (ii) can be shared among multiple robots with different embodiments to generate new obstacle maps on-the-fly (by using a list of obstacle categories). Extensive experiments carried out in simulated and real world environments show that VLMaps enable navigation according to more complex language instructions than existing methods. Videos are available at https://vlmaps.github.io.

Key facts. Cited by 365 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 50 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

39 of 50 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  12. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
27 more
  1. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  2. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  3. 2022Language-driven Semantic Segmentationcited by 717
  4. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  5. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  6. 2018On Evaluation of Embodied Navigation Agentscited by 547
  7. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  8. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  9. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  10. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  11. 2022Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimizationcited by 178
  12. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 163
  13. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  14. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 138
  15. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 135
  16. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 132
  17. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 130
  18. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 129
  19. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 107
  20. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  21. 2018Fusion++: Volumetric Object-Level SLAMcited by 98
  22. 2018MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objectscited by 95
  23. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 89
  24. 2018MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAMcited by 65
  25. 2022Affordance Learning from Play for Sample-Efficient Policy Learningcited by 30
  26. 2022Implicit Kinematic Policies: Unifying Joint and Cartesian Action Spaces in End-to-End Robot Learningcited by 15
  27. 2021Spatial Intention Maps for Multi-Agent Mobile Manipulationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 59; 2024: 127; 2025: 97; 2026: 81.

365 papers.

Sort:

By area:

  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  3. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  4. 2023A Comprehensive Overview of Large Language Modelscited by 648
  5. 2023LERF: Language Embedded Radiance Fieldscited by 543
  6. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 484
  7. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  8. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  9. 2023LangSplat: 3D Language Gaussian Splattingcited by 415
  10. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 346
  11. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  12. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
38 more
  1. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  2. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  3. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 196
  4. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  5. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  6. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 157
  7. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 156
  8. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  9. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  10. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 124
  11. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 110
  12. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  13. 2023On Bringing Robots Homecited by 101
  14. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  15. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 89
  16. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 89
  17. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  18. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  19. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 75
  20. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
  21. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  22. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 67
  23. 20253D and 4D World Modeling: A Surveycited by 67
  24. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 65
  25. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 63
  26. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  27. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 58
  28. 2024MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentationcited by 56
  29. 2024Benchmarking Mobile Device Control Agents across Diverse Configurationscited by 55
  30. 2023Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representationcited by 54
  31. 2024GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplanecited by 53
  32. 2024VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Modelscited by 46
  33. 2023Prompt a Robot to Walk with Large Language Modelscited by 45
  34. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  35. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 44
  36. 2023Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptionscited by 42
  37. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 42
  38. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41

50 most cited of 365 All 365 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Do Visual-Language Grid Maps Capture Latent Semantics?
  2. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation
  3. 2025OpenMap: Instruction Grounding via Open-Vocabulary Visual-Language Mapping
  4. 2025osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning
  5. 2026GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model
  6. 2024Where to Fetch: Extracting Visual Scene Representation from Large Pre-Trained Models for Robotic Goal Navigation ↗
  7. 2026Language as a Sensor: Calibrated Spatial Belief Estimation in 3D Scenes from Natural Language ↗
  8. 2024Navigation with VLM framework: Towards Going to Any Language