# OpenFusion++: An Open-vocabulary Real-time Scene Understanding System

arXiv: https://arxiv.org/abs/2504.19266 · HTML: https://mlsanity.com/paper/2504.19266/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 28 references that are in the map.
- First posted on arXiv 27 Apr 2025.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)
- [SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2211.14813/) (2022)
- [SAM3D: Segment Anything in 3D Scenes](https://mlsanity.com/paper/2306.03908/) (2023)
- [Alpha-CLIP: A CLIP Model Focusing on Wherever You Want](https://mlsanity.com/paper/2312.03818/) (2023)
- [Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance](https://mlsanity.com/paper/2312.10671/) (2023)
- [OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation](https://mlsanity.com/paper/2309.00616/) (2023)
- [MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2401.07745/) (2024)
- [Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representation](https://mlsanity.com/paper/2310.03923/) (2023)
- [EmbodiedSAM: Online Segment Any 3D Thing in Real Time](https://mlsanity.com/paper/2408.11811/) (2024)
- [OpenSU3D: Open World 3D Scene Understanding using Foundation Models](https://mlsanity.com/paper/2407.14279/) (2024)
- [Towards Real-Time Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2412.04434/) (2024)
- [PanoSLAM: Panoptic 3D Scene Reconstruction via Gaussian SLAM](https://mlsanity.com/paper/2501.00352/) (2024)

## Built on by (most cited first, 3 of 3)

- [IRIS-SLAM: Unified Geo-Instance Representations for Robust Semantic Localization and Mapping](https://mlsanity.com/paper/2602.18709/) (2026)
- [LatentAM: Real-Time, Large-Scale Latent Gaussian Attention Mapping via Online Dictionary Learning](https://mlsanity.com/paper/2602.12314/) (2026)
- [3DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answering](https://mlsanity.com/paper/2608.01185/) (2026)

## Abstract (arXiv)

Real-time open-vocabulary scene understanding is essential for efficient 3D perception in applications such as vision-language navigation, embodied intelligence, and augmented reality. However, existing methods suffer from imprecise instance segmentation, static semantic updates, and limited handling of complex queries. To address these issues, we present OpenFusion++, a TSDF-based real-time 3D semantic-geometric reconstruction system. Our approach refines 3D point clouds by fusing confidence maps from foundational models, dynamically updates global semantic labels via an adaptive cache based on instance area, and employs a dual-path encoding framework that integrates object attributes with environmental context for precise query responses. Experiments on the ICL, Replica, ScanNet, and ScanNet++ datasets demonstrate that OpenFusion++ significantly outperforms the baseline in both semantic accuracy and query responsiveness.