mlsanityData to 14 Oct 2026

Open-vocabulary Queryable Scene Representations for Real World Planning

arXiv 2209.09874 ↗First posted 20 Sep 2022cs.RO cs.AI cs.CVcited by 160

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have unlocked new capabilities of task planning from human instructions. However, prior attempts to apply LLMs to real-world robotic tasks are limited by the lack of grounding in the surrounding scene. In this paper, we develop NLMap, an open-vocabulary and queryable scene representation to address this problem. NLMap serves as a framework to gather and integrate contextual information into LLM planners, allowing them to see and query available objects in the scene before generating a context-conditioned plan. NLMap first establishes a natural language queryable scene representation with Visual Language models (VLMs). An LLM based object proposal module parses instructions and proposes involved objects to query the scene representation for object availability and location. An LLM planner then plans with such information about the scene. NLMap allows robots to operate without a fixed list of objects nor executable options, enabling real robot operation unachievable by previous methods. Project website: https://nlmap-saycan.github.io

Key facts. Cited by 160 papers in the map (data through 14 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 20 Sep 2022.

Builds on

29 of 45 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  6. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  7. 2018On Evaluation of Embodied Navigation Agentscited by 547
  8. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  9. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  10. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  11. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  12. 2020Learning to Explore using Active Neural SLAMcited by 316
17 more
  1. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  2. 2020Neural Topological SLAM for Visual Navigationcited by 195
  3. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 178
  4. 2017PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimationcited by 170
  5. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 163
  6. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 150
  7. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 150
  8. 2020ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votescited by 119
  9. 2019Learning Exploration Policies for Navigationcited by 116
  10. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 116
  11. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 107
  12. 2018Fusion++: Volumetric Object-Level SLAMcited by 98
  13. 2018MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objectscited by 95
  14. 2018Visual Representations for Semantic Target Driven Navigationcited by 89
  15. 2020Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphscited by 83
  16. 2019A Behavioral Approach to Visual Navigation with Graph Localization Networkscited by 40
  17. 2020Learning Object-conditioned Exploration using Distributed Soft Actor Criticcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 44; 2024: 67; 2025: 33; 2026: 10.

160 papers.

Sort:

By area:

  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  3. 2023LERF: Language Embedded Radiance Fieldscited by 543
  4. 2023Cognitive Architectures for Language Agentscited by 429
  5. 2023LangSplat: 3D Language Gaussian Splattingcited by 415
  6. 2023Augmented Language Models: a Surveycited by 403
  7. 2022Visual Language Maps for Robot Navigationcited by 365
  8. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  9. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  10. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  11. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  12. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 224
38 more
  1. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  2. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 196
  3. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  4. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  5. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  6. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 165
  7. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 156
  8. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 153
  9. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  10. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 133
  11. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 129
  12. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 127
  13. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 109
  14. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 106
  15. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  16. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  17. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 89
  18. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 89
  19. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  20. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  21. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 73
  22. 2023Making Language Models Better Tool Learners with Execution Feedbackcited by 70
  23. 2023Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agentscited by 69
  24. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 67
  25. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  26. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  27. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 58
  28. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 57
  29. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 55
  30. 2023Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representationcited by 54
  31. 2024GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplanecited by 53
  32. 2024LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agentscited by 51
  33. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 50
  34. 2024Physical Property Understanding from Language-Embedded Feature Fieldscited by 48
  35. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  36. 2023Large Language Models Are Semi-Parametric Reinforcement Learning Agentscited by 42
  37. 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
  38. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Relational Scene Graphs for Object Grounding of Natural Language Commands
  2. 2023Task and Motion Planning with Large Language Models for Object Rearrangement
  3. 2024Where to Fetch: Extracting Visual Scene Representation from Large Pre-Trained Models for Robotic Goal Navigation ↗
  4. 2025Assessing the Value of Visual Input: A Benchmark of Multimodal Large Language Models for Robotic Path Planning
  5. 2025Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarks
  6. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
  7. 2025osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning
  8. 2024CLMASP: Coupling Large Language Models with Answer Set Programming for Robotic Task Planning