mlsanityData to 14 Oct 2026

Language-Grounded Hierarchical Planning and Execution with Multi-Robot 3D Scene Graphs

arXiv 2506.07454 ↗First posted 9 Jun 2025cs.RO cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a multi-robot system that integrates mapping, localization, and task and motion planning (TAMP) enabled by 3D scene graphs to execute complex instructions expressed in natural language. Our system builds a shared 3D scene graph incorporating an open-set object-based map, which is leveraged for multi-robot 3D scene graph fusion. This representation supports real-time, view-invariant relocalization (via the object-based map) and planning (via the 3D scene graph), allowing a team of robots to reason about their surroundings and execute complex tasks. Additionally, we introduce a planning approach that translates operator intent into Planning Domain Definition Language (PDDL) goals using a Large Language Model (LLM) by leveraging context from the shared 3D scene graph and robot capabilities. We provide an experimental assessment of the performance of our system on real-world tasks in large-scale, outdoor environments. A supplementary video is available at https://youtu.be/8xbGGOLfLAY.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 30 of its 38 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

30 of 38 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  3. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  4. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,320
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  6. 2023LightGlue: Local Feature Matching at Light Speedcited by 590
  7. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  8. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  9. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 294
  10. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  11. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 196
  12. 2023Optimal Transport Aggregation for Visual Place Recognitioncited by 152
18 more
  1. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 150
  2. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  3. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 133
  4. 2023AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkerscited by 113
  5. 2024Clio: Real-time Task-Driven Open-Set 3D Scene Graphscited by 81
  6. 2023Foundations of Spatial Perception for Robotics: Hierarchical Representations and Real-time Systemscited by 60
  7. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  8. 2017X-View: Graph-Based Semantic Multi-View Localizationcited by 43
  9. 2022S-Graphs+: Real-time Localization and Mapping leveraging Hierarchical Representationscited by 28
  10. 2023Optimal Scene Graph Planning with Large Language Model Guidancecited by 27
  11. 2023Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologiescited by 25
  12. 2023Hydra-Multi: Collaborative Online Construction of 3D Scene Graphs with Multi-Robot Teamscited by 21
  13. 2023Resilient and Distributed Multi-Robot Visual SLAM: Datasets, Experiments, and Lessons Learnedcited by 16
  14. 2024Task and Motion Planning in Hierarchical 3D Scene Graphscited by 12
  15. 2024SlideSLAM: Sparse, Lightweight, Decentralized Metric-Semantic SLAM for Multi-Robot Navigationcited by 11
  16. 2024ROMAN: Open-Set Object Map Alignment for Robust View-Invariant Global Localizationcited by 10
  17. 2024GOReloc: Graph-based Object-Level Relocalization for Visual SLAMcited by 7
  18. 2024CLIPPER: Robust Data Association without an Initial Guesscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025RAVEN: Resilient Aerial Navigation via Open-Set Semantic Memory and Behavior Adaptationcited by 10
  2. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  3. 2025Structured Interfaces for Automated Reasoning with 3D Scene Graphscited by 3
  4. 2025Heterogeneous Robot Collaboration in Unstructured Environments with Grounded Generative Intelligencecited by 3
  5. 2025Prompting Robot Teams with Natural Languagecited by 0
  6. 2025CoFineLLM: Conformal Finetuning of LLMs for Language-Instructed Robot Planningcited by 0