# Hierarchical Aggregation of Semantic Uncertainty in 3D Scene Graphs

arXiv: https://arxiv.org/abs/2609.22351 · HTML: https://mlsanity.com/paper/2609.22351/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [Delving into Out-of-Distribution Detection with Vision-Language Representations](https://mlsanity.com/paper/2211.13445/) (2022)
- [Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation](https://mlsanity.com/paper/2403.17846/) (2024)
- [Habitat-Matterport 3D Semantics Dataset](https://mlsanity.com/paper/2210.05633/) (2022)
- [Negative Label Guided OOD Detection with Pretrained Vision-Language Models](https://mlsanity.com/paper/2403.20078/) (2024)
- [OpenGraph: Open-Vocabulary Hierarchical 3D Graph Representation in Large-Scale Outdoor Environments](https://mlsanity.com/paper/2403.09412/) (2024)
- [Describe Anything Anywhere At Any Moment](https://mlsanity.com/paper/2512.00565/) (2025)
- [Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume](https://mlsanity.com/paper/2602.24195/) (2026)
- [On the Overconfidence Problem in Semantic 3D Mapping](https://mlsanity.com/paper/2311.10018/) (2023)
- [Relationship-Aware Hierarchical 3D Scene Graph for Task Reasoning](https://mlsanity.com/paper/2602.02456/) (2026)
- [Belief Scene Graphs: Expanding Partial Scenes with Objects through Computation of Expectation](https://mlsanity.com/paper/2402.03840/) (2024)
- [Generation of Uncertainty-Aware High-Level Spatial Concepts in Factorized 3D Scene Graphs via Graph Neural Networks](https://mlsanity.com/paper/2409.11972/) (2024)
- [Estimating Commonsense Scene Composition on Belief Scene Graphs](https://mlsanity.com/paper/2505.02405/) (2025)
- [Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guarantees](https://mlsanity.com/paper/2606.08277/) (2026)
- [Occupancy-Grounded Room Segmentation for Hierarchical 3D Scene Graphs](https://mlsanity.com/paper/2606.13727/) (2026)
- [FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty](https://mlsanity.com/paper/2606.14728/) (2026)

## Abstract (arXiv)

Open-vocabulary 3D Scene Graphs (3DSGs) ground each object node in a vision-language embedding, yet they record every entry as equally certain, so a robot querying the map cannot tell which of its entries are unreliable. Estimators of semantic uncertainty could supply that distinction, but they require repeated sampling of a model, training, or held-out labels, none of which are available to a deployed system at query time. We present a framework that exploits the detector confidence and the embeddings a 3DSG already stores, converts them into a probability that an entry is correct, and propagates that probability through the containment hierarchy into a belief that a room contains a queried class. Four signals, each paired with the object-level error it indicates, are converted to probabilities at the logit scale learned by the vision-language model and combined in closed form with no additional perception or training. Objects sharing a detector and a vocabulary fail together, so the framework aggregates them in the fully correlated limit, where an aggregation under independence would treat one repeated error as repeated evidence. Evaluated on HM3DSem against a state-of-the-art 3DSG system, the framework improves object retrieval and lowers the error of the room-level assertions of the graph it reads.