# O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit Representation

arXiv: https://arxiv.org/abs/2404.06836 · HTML: https://mlsanity.com/paper/2404.06836/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 10 Apr 2024.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes](https://mlsanity.com/paper/2101.10994/) (2021)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023)
- [Side Adapter Network for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2302.12242/) (2023)
- [Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP](https://mlsanity.com/paper/2308.02487/) (2023)
- [A Simple Framework for Open-Vocabulary Segmentation and Detection](https://mlsanity.com/paper/2303.08131/) (2023)
- [Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations](https://mlsanity.com/paper/2209.03494/) (2022)
- [SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2211.14813/) (2022)
- [SAM3D: Segment Anything in 3D Scenes](https://mlsanity.com/paper/2306.03908/) (2023)
- [Neural Importance Sampling](https://mlsanity.com/paper/1808.03856/) (2018)
- [Weakly Supervised 3D Open-vocabulary Segmentation](https://mlsanity.com/paper/2305.14093/) (2023)
- [Neural 3D Scene Reconstruction with the Manhattan-world Assumption](https://mlsanity.com/paper/2205.02836/) (2022)
- [CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction](https://mlsanity.com/paper/2310.01403/) (2023)
- [ReLU Fields: The Little Non-linearity That Could](https://mlsanity.com/paper/2205.10824/) (2022)
- [SA-ConvONet: Sign-Agnostic Optimization of Convolutional Occupancy Networks](https://mlsanity.com/paper/2105.03582/) (2021)

## Built on by (most cited first, 9 of 9)

- [Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graph](https://mlsanity.com/paper/2406.07113/) (2024)
- [Open-Vocabulary Online Semantic Mapping for SLAM](https://mlsanity.com/paper/2411.15043/) (2024)
- [OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Merging](https://mlsanity.com/paper/2503.01309/) (2025)
- [Ov3R: Open-Vocabulary Semantic 3D Reconstruction from RGB Videos](https://mlsanity.com/paper/2507.22052/) (2025)
- [OpenGS-Fusion: Open-Vocabulary Dense Mapping with Hybrid 3D Gaussian Splatting for Refined Object-Level Understanding](https://mlsanity.com/paper/2508.01150/) (2025)
- [The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Models](https://mlsanity.com/paper/2601.08876/) (2026)
- [OnlinePG: Online Open-Vocabulary Panoptic Mapping with 3D Gaussian Splatting](https://mlsanity.com/paper/2603.18510/) (2026)
- [DR-BFR: Degradation Representation with Diffusion Models for Blind Face Restoration](https://mlsanity.com/paper/2411.10508/) (2024)
- [Stream3Dv2: Geometric-Semantic Fusion Enhanced Streaming Zero-Shot 3D Scene Understanding](https://mlsanity.com/paper/2608.21136/) (2026)

## Abstract (arXiv)

Online construction of open-ended language scenes is crucial for robotic applications, where open-vocabulary interactive scene understanding is required. Recently, neural implicit representation has provided a promising direction for online interactive mapping. However, implementing open-vocabulary scene understanding capability into online neural implicit mapping still faces three challenges: lack of local scene updating ability, blurry spatial hierarchical semantic segmentation and difficulty in maintaining multi-view consistency. To this end, we proposed O2V-mapping, which utilizes voxel-based language and geometric features to create an open-vocabulary field, thus allowing for local updates during online training process. Additionally, we leverage a foundational model for image segmentation to extract language features on object-level entities, achieving clear segmentation boundaries and hierarchical semantic features. For the purpose of preserving consistency in 3D object properties across different viewpoints, we propose a spatial adaptive voxel adjustment mechanism and a multi-view weight selection method. Extensive experiments on open-vocabulary object localization and semantic segmentation demonstrate that O2V-mapping achieves online construction of language scenes while enhancing accuracy, outperforming the previous SOTA method.