mlsanityData to 7 Oct 2026

GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering

arXiv 2412.14480 ↗First posted 19 Dec 2024cs.RO cs.CL cs.CVcited by 37

Not on the map. Why?

Abstract (arXiv)

In Embodied Question Answering (EQA), agents must explore and develop a semantic understanding of an unseen environment to answer a situated question with confidence. This problem remains challenging in robotics, due to the difficulties in obtaining useful semantic representations, updating these representations online, and leveraging prior world knowledge for efficient planning and exploration. To address these limitations, we propose GraphEQA, a novel approach that utilizes real-time 3D metric-semantic scene graphs (3DSGs) and task relevant images as multi-modal memory for grounding Vision-Language Models (VLMs) to perform EQA tasks in unseen environments. We employ a hierarchical planning approach that exploits the hierarchical nature of 3DSGs for structured planning and semantics-guided exploration. We evaluate GraphEQA in simulation on two benchmark datasets, HM-EQA and OpenEQA, and demonstrate that it outperforms key baselines by completing EQA tasks with higher success rates and fewer planning steps. We further demonstrate GraphEQA in multiple real-world home and office environments.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 56 of its 63 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

56 of 63 references.

References · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Segment Anythingcited by 8,158
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  7. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  8. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  9. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  10. 2023LERF: Language Embedded Radiance Fieldscited by 538
  11. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  12. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
38 more
  1. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  2. 2022Visual Language Maps for Robot Navigationcited by 362
  3. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  4. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  5. 2017Embodied Question Answeringcited by 335
  6. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  7. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  8. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  9. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  10. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
  11. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  12. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  13. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  14. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  15. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  16. 2022Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimizationcited by 175
  17. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  18. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 159
  19. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  20. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  21. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  22. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  23. 2022Habitat-Matterport 3D Semantics Datasetcited by 109
  24. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  25. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  26. 2023GOAT: GO to Any Thingcited by 89
  27. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
  28. 2024Clio: Real-time Task-Driven Open-Set 3D Scene Graphscited by 81
  29. 20193-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agentscited by 74
  30. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  31. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 72
  32. 2024Language-Grounded Dynamic Scene Graphs for Interactive Object Search with Mobile Manipulationcited by 69
  33. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  34. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  35. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  36. 2023SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environmentscited by 53
  37. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  38. 2023Collaborative Dynamic 3D Scene Graphs for Automated Drivingcited by 34

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 22.

37 papers.

Sort:

By area:

  1. 2025Beyond the Destination: A Novel Benchmark for Exploration-Aware Embodied Question Answeringcited by 29
  2. 2025Describe Anything Anywhere At Any Momentcited by 19
  3. 2025Memory-Centric Embodied Question Answeringcited by 12
  4. 2025Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answeringcited by 12
  5. 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation Systemcited by 10
  6. 2025Meta-Memory: Retrieving and Integrating Semantic-Spatial Memories for Robot Spatial Reasoningcited by 9
  7. 2026NavWAM: A Navigation World Action Model for Goal-Conditioned Visual Navigationcited by 7
  8. 2025STRIVE: Structured Representation Integrating VLM Reasoning for Efficient Object Navigationcited by 5
  9. 2025Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Explorationcited by 5
  10. 2025Toward Ambulatory Vision: Learning Visually-Grounded Active View Selectioncited by 5
  11. 2026RieMind: Geometry-Grounded Spatial Agent for Scene Understandingcited by 4
  12. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
25 more
  1. 2025GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answeringcited by 3
  2. 2025Multi-Step Reasoning for Embodied Question Answering via Tool Augmentationcited by 3
  3. 2026HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answeringcited by 3
  4. 2026Stop Wandering: Efficient Vision-Language Navigation via Metacognitive Reasoningcited by 3
  5. 2026E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenescited by 3
  6. 2026Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guaranteescited by 3
  7. 2025Prune-Then-Plan: Step-Level Calibration for Stable Frontier Exploration in Embodied Question Answeringcited by 2
  8. 2025R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Spacecited by 2
  9. 2026Offload or Overload: A Platform Measurement Study of Mobile Robotic Manipulation Workloadscited by 2
  10. 2026Towards Next-Generation Healthcare: A Survey of Medical Embodied AI for Perception, Decision-Making, and Actioncited by 2
  11. 2025M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environmentscited by 1
  12. 2025SpatialReasoner: Active Perception for Large-Scale 3D Scene Understandingcited by 1
  13. 2026Memory-Guided View Refinement for Dynamic Human-in-the-loop EQAcited by 1
  14. 2026Meanings and Measurements: Multi-Agent Probabilistic Grounding for Vision-Language Navigationcited by 1
  15. 2026ObsGraph: Hierarchical Observation Representation for Embodied Reasoning and Explorationcited by 1
  16. 2026Automating the Design of Embodied Agent Architecturescited by 1
  17. 2026DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priorscited by 1
  18. 2026Scaffolding Foundation Models into Physical-World Agents Pushes the Frontier of Long-Horizon Navigationcited by 1
  19. 2025A Community-driven vision for a new Knowledge Resource for AIcited by 0
  20. 2026Semantic Enrichment of CAD-Based Industrial Environments via Scene Graphs for Simulation and Reasoningcited by 0
  21. 2026Active Spatial Inspection for Effective and Efficient Embodied Explorationcited by 0
  22. 2026MVVBench: Benchmarking 4D Reasoning in Vision-Language Modelscited by 0
  23. 2026PORTER: Edge-Cloud Residency for Persistent 3D Scene Graph Memorycited by 0
  24. 2026SafeVantage: Vantage-Aware Memory for Reliable Embodied Decisionscited by 0
  25. 2026NavGPT-3: Harnessing Context in a Hierarchical Navigation Runtimecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Hierarchical Floorplan-Guided Vision-Language Exploration for Embodied Question Answering ↗
  2. 2024EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answering
  3. 2022SQA3D: Situated Question Answering in 3D Scenes
  4. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering
  5. 2026FAST-EQA: Efficient Embodied Question Answering with Global and Local Region Relevancy
  6. 2024EmbodiedRAG: Dynamic 3D Scene Graph Retrieval for Efficient and Scalable Robot Task Planning
  7. 2026Extending Embodied Question Answering from Perception to Decision
  8. 20253D-MoRe: Unified Modal-Contextual Reasoning for Embodied Question Answering