# Foundations of Spatial Perception for Robotics: Hierarchical Representations and Real-time Systems

arXiv: https://arxiv.org/abs/2305.07154 · HTML: https://mlsanity.com/paper/2305.07154/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2023.

## Built on by (most cited first, 30 of 60)

- [RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation](https://mlsanity.com/paper/2402.15487/) (2024)
- [Safety Guardrails for LLM-Enabled Robots](https://mlsanity.com/paper/2503.07885/) (2025)
- [Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologies](https://mlsanity.com/paper/2312.11713/) (2023)
- [FunGraph: Functionality Aware 3D Scene Graphs for Language-Prompted Scene Interaction](https://mlsanity.com/paper/2503.07909/) (2025)
- [Describe Anything Anywhere At Any Moment](https://mlsanity.com/paper/2512.00565/) (2025)
- [SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environments](https://mlsanity.com/paper/2410.03035/) (2024)
- [Language-EXtended Indoor SLAM (LEXIS): A Versatile System for Real-time Visual Scene Understanding](https://mlsanity.com/paper/2309.15065/) (2023)
- [The Bare Necessities: Designing Simple, Effective Open-Vocabulary Scene Graphs](https://mlsanity.com/paper/2412.01539/) (2024)
- [Air-Ground Collaboration for Language-Specified Missions in Unknown Environments](https://mlsanity.com/paper/2505.09108/) (2025)
- [MCN-SLAM: Multi-Agent Collaborative Neural SLAM with Hybrid Implicit Neural Scene Representation](https://mlsanity.com/paper/2506.18678/) (2025)
- [PINGS: Gaussian Splatting Meets Distance Fields within a Point-Based Implicit Neural Map](https://mlsanity.com/paper/2502.05752/) (2025)
- [KeySG: Hierarchical Keyframe-Based 3D Scene Graphs](https://mlsanity.com/paper/2510.01049/) (2025)
- [MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planning](https://mlsanity.com/paper/2512.16909/) (2025)
- [ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis](https://mlsanity.com/paper/2504.06553/) (2025)
- [SG-Reg: Generalizable and Efficient Scene Graph Registration](https://mlsanity.com/paper/2504.14440/) (2025)
- [CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity](https://mlsanity.com/paper/2506.16652/) (2025)
- [Language-Grounded Hierarchical Planning and Execution with Multi-Robot 3D Scene Graphs](https://mlsanity.com/paper/2506.07454/) (2025)
- [GALA: Guided Attention with Language Alignment for Open Vocabulary Gaussian Splatting](https://mlsanity.com/paper/2508.14278/) (2025)
- [S-Graphs 2.0 -- A Hierarchical-Semantic Optimization and Loop Closure for SLAM](https://mlsanity.com/paper/2502.18044/) (2025)
- [Social 3D Scene Graphs: Modeling Human Actions and Relations for Interactive Service Robots](https://mlsanity.com/paper/2509.24966/) (2025)
- [Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Exploration](https://mlsanity.com/paper/2512.02458/) (2025)
- [Aion: Towards Hierarchical 4D Scene Graphs with Temporal Flow Dynamics](https://mlsanity.com/paper/2512.11903/) (2025)
- [GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoning](https://mlsanity.com/paper/2603.19137/) (2026)
- [Learning High-level Semantic-Relational Concepts for SLAM](https://mlsanity.com/paper/2310.00401/) (2023)
- [Point2Graph: An End-to-end Point Cloud-based 3D Open-Vocabulary Scene Graph for Robot Navigation](https://mlsanity.com/paper/2409.10350/) (2024)
- [LiteVLoc: Map-Lite Visual Localization for Image Goal Navigation](https://mlsanity.com/paper/2410.04419/) (2024)
- [CuriousBot: Interactive Mobile Exploration via Actionable 3D Relational Object Graph](https://mlsanity.com/paper/2501.13338/) (2025)
- [VL-Nav: Neuro-Symbolic Reasoning-based Vision-Language Navigation](https://mlsanity.com/paper/2502.00931/) (2025)
- [Generation of Uncertainty-Aware High-Level Spatial Concepts in Factorized 3D Scene Graphs via Graph Neural Networks](https://mlsanity.com/paper/2409.11972/) (2024)
- [OpenFrontier: General Navigation with Visual-Language Grounded Frontiers](https://mlsanity.com/paper/2603.05377/) (2026)

## Abstract (arXiv)

3D spatial perception is the problem of building and maintaining an actionable and persistent representation of the environment in real-time using sensor data and prior knowledge. Despite the fast-paced progress in robot perception, most existing methods either build purely geometric maps (as in traditional SLAM) or flat metric-semantic maps that do not scale to large environments or large dictionaries of semantic labels. The first part of this paper is concerned with representations: we show that scalable representations for spatial perception need to be hierarchical in nature. Hierarchical representations are efficient to store, and lead to layered graphs with small treewidth, which enable provably efficient inference. We then introduce an example of hierarchical representation for indoor environments, namely a 3D scene graph, and discuss its structure and properties. The second part of the paper focuses on algorithms to incrementally construct a 3D scene graph as the robot explores the environment. Our algorithms combine 3D geometry, topology (to cluster the places into rooms), and geometric deep learning (e.g., to classify the type of rooms the robot is moving across). The third part of the paper focuses on algorithms to maintain and correct 3D scene graphs during long-term operation. We propose hierarchical descriptors for loop closure detection and describe how to correct a scene graph in response to loop closures, by solving a 3D scene graph optimization problem. We conclude the paper by combining the proposed perception algorithms into Hydra, a real-time spatial perception system that builds a 3D scene graph from visual-inertial data in real-time. We showcase Hydra's performance in photo-realistic simulations and real data collected by a Clearpath Jackal robots and a Unitree A1 robot. We release an open-source implementation of Hydra at https://github.com/MIT-SPARK/Hydra.