mlsanityData to 7 Oct 2026

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

arXiv 2201.13360 ↗First posted 31 Jan 2022cs.ROcited by 175

Not on the map. Why?

Abstract (arXiv)

3D scene graphs have recently emerged as a powerful high-level representation of 3D environments. A 3D scene graph describes the environment as a layered graph where nodes represent spatial concepts at multiple levels of abstraction and edges represent relations between concepts. While 3D scene graphs can serve as an advanced "mental model" for robots, how to build such a rich representation in real-time is still uncharted territory. This paper describes a real-time Spatial Perception System, a suite of algorithms to build a 3D scene graph from sensor data in real-time. Our first contribution is to develop real-time algorithms to incrementally construct the layers of a scene graph as the robot explores the environment; these algorithms build a local Euclidean Signed Distance Function (ESDF) around the current robot location, extract a topological map of places from the ESDF, and then segment the places into rooms using an approach inspired by community-detection techniques. Our second contribution is to investigate loop closure detection and optimization in 3D scene graphs. We show that 3D scene graphs allow defining hierarchical descriptors for loop closure detection; our descriptors capture statistics across layers in the scene graph, ranging from low-level visual appearance to summary statistics about objects and places. We then propose the first algorithm to optimize a 3D scene graph in response to loop closures; our approach relies on embedded deformation graphs to simultaneously correct all layers of the scene graph. We implement the proposed Spatial Perception System into a architecture named Hydra, that combines fast early and mid-level perception processes with slower high-level perception. We evaluate Hydra on simulated and real data and show it is able to reconstruct 3D scene graphs with an accuracy comparable with batch offline methods despite running online.

Key facts. Cited by 175 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Jan 2022.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 19; 2024: 40; 2025: 39; 2026: 74.

175 papers.

Sort:

By area:

  1. 2022Visual Language Maps for Robot Navigationcited by 362
  2. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  3. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  4. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  5. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  6. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  7. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
  8. 2023CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusioncited by 54
  9. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  10. 2024SemGauss-SLAM: Dense Semantic Gaussian Splatting SLAMcited by 41
  11. 2023Incremental 3D Semantic Scene Graph Prediction from RGB Sequencescited by 39
  12. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
38 more
  1. 2024DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Modelscited by 33
  2. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  3. 2023SGAligner : 3D Scene Alignment with Scene Graphscited by 30
  4. 2024SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphscited by 29
  5. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  6. 2022S-Graphs+: Real-time Localization and Mapping leveraging Hierarchical Representationscited by 28
  7. 2024EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agentscited by 24
  8. 2024General Place Recognition Survey: Towards Real-World Autonomycited by 23
  9. 2025RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Explorationcited by 23
  10. 2024OpenGraph: Open-Vocabulary Hierarchical 3D Graph Representation in Large-Scale Outdoor Environmentscited by 22
  11. 2025FunGraph: Functionality Aware 3D Scene Graphs for Language-Prompted Scene Interactioncited by 22
  12. 2023SGRec3D: Self-Supervised 3D Scene Graph Learning via Object-Level Scene Reconstructioncited by 20
  13. 2023Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph predictioncited by 18
  14. 2024RelationField: Relate Anything in Radiance Fieldscited by 16
  15. 2023Language-EXtended Indoor SLAM (LEXIS): A Versatile System for Real-time Visual Scene Understandingcited by 15
  16. 2024Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Modelscited by 12
  17. 2023ConvBKI: Real-Time Probabilistic Semantic Mapping Network with Quantifiable Uncertaintycited by 10
  18. 2024SG-PGM: Partial Graph Matching Network with Semantic Geometric Fusion for 3D Scene Graph Alignment and Its Downstream Taskscited by 10
  19. 2024Lost & Found: Tracking Changes from Egocentric Observations in 3D Dynamic Scene Graphscited by 10
  20. 2025FROSS: Faster-than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Imagescited by 10
  21. 2023DRAGON: A Dialogue-Based Robot for Assistive Navigation with Visual Language Groundingcited by 9
  22. 2024External Knowledge Enhanced 3D Scene Generation from Sketchcited by 9
  23. 2024Multiview Scene Graphcited by 9
  24. 2025Hierarchical Planning for Complex Tasks with Knowledge Graph-RAG and Symbolic Verificationcited by 9
  25. 2025MORE: Mobile Manipulation Rearrangement Through Grounded Language Reasoningcited by 9
  26. 2025Mobi-π: Mobilizing Your Robot Learning Policycited by 9
  27. 2025Meta-Memory: Retrieving and Integrating Semantic-Spatial Memories for Robot Spatial Reasoningcited by 9
  28. 2026Articulated 3D Scene Graphs for Open-World Mobile Manipulationcited by 9
  29. 2026SimRecon: SimReady Compositional Scene Reconstruction from Real Videoscited by 9
  30. 2024Embodied Instruction Following in Unknown Environmentscited by 8
  31. 2025IRef-VLA: A Benchmark for Interactive Referential Grounding with Imperfect Language in 3D Scenescited by 8
  32. 2025SG-Reg: Generalizable and Efficient Scene Graph Registrationcited by 8
  33. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguitycited by 8
  34. 2025KeySG: Hierarchical Keyframe-Based 3D Scene Graphscited by 8
  35. 2023Neural Implicit Vision-Language Feature Fieldscited by 7
  36. 2024CAMON: Cooperative Agents for Multi-Object Navigation with LLM-based Conversationscited by 7
  37. 2024GV-Bench: Benchmarking Local Feature Matching for Geometric Verification of Long-term Loop Closure Detectioncited by 7
  38. 2024Scene-Graph ViT: End-to-End Open-Vocabulary Visual Relationship Detectioncited by 6

50 most cited of 175 All 175 (JSON)