mlsanityData to 14 Oct 2026

Open-Vocabulary Online Semantic Mapping for SLAM

arXiv 2411.15043 ↗First posted 22 Nov 2024cs.CV cs.ROcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper presents an Open-Vocabulary Online 3D semantic mapping pipeline, that we denote by its acronym OVO. Given a sequence of posed RGB-D frames, we detect and track 3D segments, which we describe using CLIP vectors. These are computed from the viewpoints where they are observed by a novel CLIP merging method. Notably, our OVO has a significantly lower computational and memory footprint than offline baselines, while also showing better segmentation metrics than offline and online ones. Along with superior segmentation performance, we also show experimental results of our mapping contributions integrated with two different full SLAM backbones (Gaussian-SLAM and ORB-SLAM2), being the first ones using a neural network to merge CLIP descriptors and demonstrating end-to-end open-vocabulary online 3D mapping with loop closure.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 60 of its 77 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

60 of 77 references.

References · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Segment Anythingcited by 8,234
  3. 2017Mask R-CNNcited by 7,277
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  9. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,320
  10. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  11. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  12. 2020ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAMcited by 878
38 more
  1. 2016ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Camerascited by 806
  2. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  3. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  4. 2021NICE-SLAM: Neural Implicit Scalable Encoding for SLAMcited by 589
  5. 2021Extract Free Dense Labels from CLIPcited by 583
  6. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  7. 2023Segment Everything Everywhere All at Oncecited by 548
  8. 2023LERF: Language Embedded Radiance Fieldscited by 543
  9. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  10. 2017VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimatorcited by 423
  11. 2023LangSplat: 3D Language Gaussian Splattingcited by 415
  12. 2016Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Agecited by 411
  13. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  14. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  15. 2023Fast Segment Anythingcited by 290
  16. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  17. 2023Data Filtering Networkscited by 250
  18. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  19. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  20. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
  21. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 196
  22. 2019Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mappingcited by 184
  23. 2023Segment Any 3D Gaussianscited by 161
  24. 2023Gaussian-SLAM: Photo-realistic Dense SLAM with Gaussian Splattingcited by 154
  25. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 151
  26. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 135
  27. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 124
  28. 2018CubeSLAM: Monocular 3D Object SLAMcited by 102
  29. 2021Large-scale Unsupervised Semantic Segmentationcited by 101
  30. 2021SuMa++: Efficient LiDAR-based Semantic SLAMcited by 100
  31. 2018Fusion++: Volumetric Object-Level SLAMcited by 98
  32. 2019PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Thingscited by 98
  33. 2024SGS-SLAM: Semantic Gaussian Splatting For Neural Dense SLAMcited by 87
  34. 2018QuadricSLAM: Dual Quadrics from Object Detections as Landmarks in Object-oriented SLAMcited by 86
  35. 2024How NeRFs and 3D Gaussian Splatting are Reshaping SLAM: a Surveycited by 72
  36. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 69
  37. 2023SNI-SLAM: Semantic Neural Implicit SLAMcited by 65
  38. 2023Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representationcited by 54

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 11.

12 papers.

  1. 2026FeatureSLAM: Feature-enriched 3D gaussian splatting SLAM in real timecited by 5
  2. 2025GS4: Generalizable Sparse Splatting Semantic SLAMcited by 2
  3. 2026OnlinePG: Online Open-Vocabulary Panoptic Mapping with 3D Gaussian Splattingcited by 2
  4. 2026IRIS-SLAM: Unified Geo-Instance Representations for Robust Semantic Localization and Mappingcited by 1
  5. 2026FUS3DMaps: Scalable and Accurate Open-Vocabulary Semantic Mapping by 3D Fusion of Voxel- and Instance-Level Layerscited by 1
  6. 2026HoloAgent-0: A Unified Embodied Agent Framework with 3D Spatial Memorycited by 1
  7. 2026Cross-Attentive Multiview Fusion of Vision-Language Embeddingscited by 0
  8. 2026Robotic Contextual Awareness for Human-Robot Collaboration and Environmental Understandingcited by 0
  9. 2026VOIM: Training-Free Open-Vocabulary 3D Instance Mapping for RGB-D and Monocular SLAMcited by 0
  10. 2026Finder: Agentic Closed-Loop Object Finding for Embodied Groundingcited by 0
  11. 2026PerSeM: Persistent Semantic Memory for Long-Horizon Open-Vocabulary UAV Mappingcited by 0
  12. 2026TRACKGRAPH: Online Open-Vocabulary 3D Scene Graphs via Image-Space Trackingcited by 0