# QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understanding

arXiv: https://arxiv.org/abs/2404.06442 · HTML: https://mlsanity.com/paper/2404.06442/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 58 references that are in the map.
- First posted on arXiv 9 Apr 2024.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [OccuSeg: Occupancy-aware 3D Instance Segmentation](https://mlsanity.com/paper/2003.06537/) (2020)
- [SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation](https://mlsanity.com/paper/1711.08588/) (2017)
- [Open-vocabulary Queryable Scene Representations for Real World Planning](https://mlsanity.com/paper/2209.09874/) (2022)
- [Unseen Object Instance Segmentation for Robotic Environments](https://mlsanity.com/paper/2007.08073/) (2020)
- [Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis](https://mlsanity.com/paper/2011.06961/) (2020)
- [Indoor Semantic Segmentation using depth information](https://mlsanity.com/paper/1301.3572/) (2013)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries](https://mlsanity.com/paper/2211.15658/) (2022)
- [Floor-SP: Inverse CAD for Floorplans by Sequential Room-wise Shortest Path](https://mlsanity.com/paper/1908.06702/) (2019)
- [FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans](https://mlsanity.com/paper/1804.00090/) (2018)
- [Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs](https://mlsanity.com/paper/2309.15940/) (2023)
- [HEAT: Holistic Edge Attention Transformer for Structured Reconstruction](https://mlsanity.com/paper/2111.15143/) (2021)
- [Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologies](https://mlsanity.com/paper/2312.11713/) (2023)
- [Situational Graphs for Robot Navigation in Structured Indoor Environments](https://mlsanity.com/paper/2202.12197/) (2022)
- [Place Categorization and Semantic Mapping on a Mobile Robot](https://mlsanity.com/paper/1507.02428/) (2015)
- [Long-Term Localization using Semantic Cues in Floor Plan Maps](https://mlsanity.com/paper/2210.01456/) (2022)
- [SceneCut: Joint Geometric and Object Segmentation for Indoor Scenes](https://mlsanity.com/paper/1709.07158/) (2017)

## Built on by (most cited first, 5 of 5)

- [Point2Graph: An End-to-end Point Cloud-based 3D Open-Vocabulary Scene Graph for Robot Navigation](https://mlsanity.com/paper/2409.10350/) (2024)
- [SUM-AgriVLN: Spatial Understanding Memory for Agricultural Vision-and-Language Navigation](https://mlsanity.com/paper/2510.14357/) (2025)
- [Open-Vocabulary Semantic Segmentation with Uncertainty Alignment for Robotic Scene Understanding in Indoor Building Environments](https://mlsanity.com/paper/2503.23105/) (2025)
- [Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomy](https://mlsanity.com/paper/2511.11777/) (2025)
- [Scene Representation using 360deg Saliency Graph and its Application in Vision-based Indoor Navigation](https://mlsanity.com/paper/2603.20353/) (2026)

## Abstract (arXiv)

Robotic tasks such as planning and navigation require a hierarchical semantic understanding of a scene, which could include multiple floors and rooms. Current methods primarily focus on object segmentation for 3D scene understanding. However, such methods struggle to segment out topological regions like "kitchen" in the scene. In this work, we introduce a two-step pipeline to solve this problem. First, we extract a topological map, i.e., floorplan of the indoor scene using a novel multi-channel occupancy representation. Then, we generate CLIP-aligned features and semantic labels for every room instance based on the objects it contains using a self-attention transformer. Our language-topology alignment supports natural language querying, e.g., a "place to cook" locates the "kitchen". We outperform the current state-of-the-art on room segmentation by ~20% and room classification by ~12%. Our detailed qualitative analysis and ablation studies provide insights into the problem of joint structural and semantic 3D scene understanding. Project Page: quest-maps.github.io