# Ov3R: Open-Vocabulary Semantic 3D Reconstruction from RGB Videos

arXiv: https://arxiv.org/abs/2507.22052 · HTML: https://mlsanity.com/paper/2507.22052/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 69 references that are in the map.
- First posted on arXiv 29 Jul 2025.

## Builds on (most cited first, 30 of 53)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images](https://mlsanity.com/paper/2403.14627/) (2024)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [Neuralangelo: High-Fidelity Neural Surface Reconstruction](https://mlsanity.com/paper/2306.03092/) (2023)
- [MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion](https://mlsanity.com/paper/2410.03825/) (2024)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision](https://mlsanity.com/paper/2410.19115/) (2024)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)
- [PGSR: Planar-based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction](https://mlsanity.com/paper/2406.06521/) (2024)
- [3D Reconstruction with Spatial Memory](https://mlsanity.com/paper/2408.16061/) (2024)
- [Pixel-Perfect Structure-from-Motion with Featuremetric Refinement](https://mlsanity.com/paper/2108.08291/) (2021)
- [Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation](https://mlsanity.com/paper/2403.17846/) (2024)
- [NICER-SLAM: Neural Implicit Scene Encoding for RGB SLAM](https://mlsanity.com/paper/2302.03594/) (2023)

## Built on by (most cited first, 10 of 10)

- [GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation](https://mlsanity.com/paper/2512.23180/) (2025)
- [Unified Semantic Transformer for 3D Scene Understanding](https://mlsanity.com/paper/2512.14364/) (2025)
- [Pano3D: Unified 3D Reconstruction and Panoptic Segmentation](https://mlsanity.com/paper/2606.14307/) (2026)
- [Genie 4D: Semantic-Prior-Guided 4D Dynamic Scene Reconstruction](https://mlsanity.com/paper/2604.09877/) (2026)
- [Cross-Attentive Multiview Fusion of Vision-Language Embeddings](https://mlsanity.com/paper/2604.12551/) (2026)
- [Bidirectional Cross-Modal Prompting for Event-Frame Asymmetric Stereo](https://mlsanity.com/paper/2604.15312/) (2026)
- [RSGPNet: Geometric Prompting for Remote Sensing Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2606.28410/) (2026)
- [MAGiSt3R: Multi-Agent Feed-forward 3D Reconstruction from Monocular RGB Videos](https://mlsanity.com/paper/2607.15211/) (2026)
- [GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generation](https://mlsanity.com/paper/2608.16234/) (2026)
- [VOIM: Training-Free Open-Vocabulary 3D Instance Mapping for RGB-D and Monocular SLAM](https://mlsanity.com/paper/2609.00775/) (2026)

## Abstract (arXiv)

We present Ov3R, a novel framework for open-vocabulary semantic 3D reconstruction from RGB video streams, designed to advance Spatial AI. The system features two key components: CLIP3R, a CLIP-informed 3D reconstruction module that predicts dense point maps from overlapping clips while embedding object-level semantics; and 2D-3D OVS, a 2D-3D open-vocabulary semantic module that lifts 2D features into 3D by learning fused descriptors integrating spatial, geometric, and semantic cues. Unlike prior methods, Ov3R incorporates CLIP semantics directly into the reconstruction process, enabling globally consistent geometry and fine-grained semantic alignment. Our framework achieves state-of-the-art performance in both dense 3D reconstruction and open-vocabulary 3D segmentation, marking a step forward toward real-time, semantics-aware Spatial AI.