mlsanityData to 7 Oct 2026

Language-Grounded Dynamic Scene Graphs for Interactive Object Search with Mobile Manipulation

arXiv 2403.08605 ↗First posted 13 Mar 2024cs.ROcited by 69

Not on the map. Why?

Abstract (arXiv)

To fully leverage the capabilities of mobile manipulation robots, it is imperative that they are able to autonomously execute long-horizon tasks in large unexplored environments. While large language models (LLMs) have shown emergent reasoning skills on arbitrary tasks, existing work primarily concentrates on explored environments, typically focusing on either navigation or manipulation tasks in isolation. In this work, we propose MoMa-LLM, a novel approach that grounds language models within structured representations derived from open-vocabulary scene graphs, dynamically updated as the environment is explored. We tightly interleave these representations with an object-centric action space. Given object detections, the resulting approach is zero-shot, open-vocabulary, and readily extendable to a spectrum of mobile manipulation and household robotic tasks. We demonstrate the effectiveness of MoMa-LLM in a novel semantic interactive search task in large realistic indoor environments. In extensive experiments in both simulation and the real world, we show substantially improved search efficiency compared to conventional baselines and state-of-the-art approaches, as well as its applicability to more abstract tasks. We make the code publicly available at http://moma-llm.cs.uni-freiburg.de.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Mar 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 29; 2026: 26.

69 papers.

Sort:

By area:

  1. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  2. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  3. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  4. 2024BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulationcited by 31
  5. 2025Safety Guardrails for LLM-Enabled Robotscited by 30
  6. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 26
  7. 2025RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Explorationcited by 23
  8. 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
  9. 2025Describe Anything Anywhere At Any Momentcited by 19
  10. 2024Open-World Amodal Appearance Completioncited by 18
  11. 2025Articulated Object Estimation in the Wildcited by 18
  12. 2024SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environmentscited by 16
38 more
  1. 2025Air-Ground Collaboration for Language-Specified Missions in Unknown Environmentscited by 11
  2. 2025Distilling On-device Language Models for Robot Planning with Minimal Human Interventioncited by 10
  3. 2025MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoningcited by 9
  4. 2026Articulated 3D Scene Graphs for Open-World Mobile Manipulationcited by 9
  5. 2025KeySG: Hierarchical Keyframe-Based 3D Scene Graphscited by 8
  6. 2025MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planningcited by 8
  7. 2025Subtask-Aware Visual Reward Learning from Segmented Demonstrationscited by 7
  8. 2026Position: Modular Memory is the Key to Continual Learning Agentscited by 7
  9. 2024Perception Matters: Enhancing Embodied AI with Uncertainty-Aware Semantic Segmentationcited by 6
  10. 2024Whole-Body Teleoperation for Mobile Manipulation at Zero Added Costcited by 6
  11. 2025GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Drivingcited by 6
  12. 2026RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generationcited by 5
  13. 2026FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoningcited by 5
  14. 2024Jailbreaking LLM-Controlled Robotscited by 4
  15. 2025CuriousBot: Interactive Mobile Exploration via Actionable 3D Relational Object Graphcited by 4
  16. 2025Long-Term Planning Around Humans in Domestic Environments with 3D Scene Graphscited by 4
  17. 2026RieMind: Geometry-Grounded Spatial Agent for Scene Understandingcited by 4
  18. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  19. 2025Deploying Foundation Model-Enabled Air and Ground Robots in the Field: Challenges and Opportunitiescited by 3
  20. 2025Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Reviewcited by 3
  21. 2025GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answeringcited by 3
  22. 2025Enhancing LLM-based Autonomous Driving with Modular Traffic Light and Sign Recognitioncited by 3
  23. 2025Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulationcited by 3
  24. 2025LookPlanGraph: Embodied Instruction Following Method with VLM Graph Augmentationcited by 3
  25. 2026Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guaranteescited by 3
  26. 2024TESGNN: Temporal Equivariant Scene Graph Neural Networks for Efficient and Robust Multi-View 3D Scene Understandingcited by 2
  27. 2026LOST-3DSG: Lightweight Open-Vocabulary 3D Scene Graphs with Semantic Tracking in Dynamic Environmentscited by 2
  28. 2026How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Studycited by 2
  29. 2026ContextFlow: Hierarchical Task-State Alignment for Long-Horizon Embodied Agentscited by 2
  30. 2024CenterArt: Joint Shape Reconstruction and 6-DoF Grasp Estimation of Articulated Objectscited by 1
  31. 2025OpenLex3D: A Tiered Evaluation Benchmark for Open-Vocabulary 3D Scene Representationscited by 1
  32. 2025M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environmentscited by 1
  33. 2025An LLM-powered Natural-to-Robotic Language Translation Framework with Correctness Guaranteescited by 1
  34. 2025Agentic Scene Policies: Unifying Space, Semantics, and Affordances for Robot Actioncited by 1
  35. 2025RecurGS: Interactive Scene Modeling via Discrete-State Recurrent Gaussian Fusioncited by 1
  36. 2026Just-In-Time Scene Graph Growth: Combating Perceptual Saturation in Long-Horizon Roboticscited by 1
  37. 2024Language-guided Robust Navigation for Mobile Robots in Dynamically-changing Environmentscited by 0
  38. 2024OV-HHIR: Open Vocabulary Human Interaction Recognition Using Cross-modal Integration of Large Language Modelscited by 0

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Language-Enhanced Mobile Manipulation for Efficient Object Search in Indoor Environments
  2. 2024Dynamic Open-Vocabulary 3D Scene Graphs for Long-term Language-Guided Mobile Manipulation
  3. 2024Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Maps
  4. 2024DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation
  5. 2025ACTLLM: Action Consistency Tuned Large Language Model
  6. 2026SG-VLA: Learning Spatially-Grounded Vision-Language-Action Models for Mobile Manipulation
  7. 2025OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints
  8. 2025Toward Accurate Long-Horizon Robotic Manipulation: Language-to-Action with Foundation Models via Scene Graphs