mlsanityData to 14 Oct 2026

GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields

arXiv 2404.00931 ↗First posted 1 Apr 2024cs.CVcited by 8

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recent advancements in vision-language foundation models have significantly enhanced open-vocabulary 3D scene understanding. However, the generalizability of existing methods is constrained due to their framework designs and their reliance on 3D data. We address this limitation by introducing Generalizable Open-Vocabulary Neural Semantic Fields (GOV-NeSF), a novel approach offering a generalizable implicit representation of 3D scenes with open-vocabulary semantics. We aggregate the geometry-aware features using a cost volume, and propose a Multi-view Joint Fusion module to aggregate multi-view features through a cross-view attention mechanism, which effectively predicts view-specific blending weights for both colors and open-vocabulary features. Remarkably, our GOV-NeSF exhibits state-of-the-art performance in both 2D and 3D open-vocabulary semantic segmentation, eliminating the need for ground truth semantic labels or depth priors, and effectively generalize across scenes and datasets without fine-tuning.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 58 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

49 of 58 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  8. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  12. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
37 more
  1. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  2. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,712
  3. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  4. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  6. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  7. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  9. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 740
  10. 2022Language-driven Semantic Segmentationcited by 717
  11. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 703
  12. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  13. 2021Extract Free Dense Labels from CLIPcited by 583
  14. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  15. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  16. 2023LERF: Language Embedded Radiance Fieldscited by 543
  17. 2022Point-NeRF: Point-based Neural Radiance Fieldscited by 503
  18. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  19. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 415
  20. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  21. 2019ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed datacited by 261
  22. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  23. 2021NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Videocited by 241
  24. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  25. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
  26. 2021Neural Rays for Occlusion-aware Image-based Renderingcited by 192
  27. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  28. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 160
  29. 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
  30. 2018ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentationcited by 133
  31. 2017Superhuman Accuracy on the SNEMI3D Connectomics Challengecited by 64
  32. 2023NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detectioncited by 51
  33. 2023Semantic Ray: Learning a Generalizable Semantic Field with Cross-Reprojection Attentioncited by 29
  34. 2023VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representationscited by 23
  35. 2018Real-time Progressive 3D Semantic Segmentation for Indoor Scenecited by 18
  36. 2023Open-NeRF: Towards Open Vocabulary NeRF Decompositioncited by 13
  37. 2023GNeSF: Generalizable Neural Semantic Fieldscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 2.

8 papers.

  1. 2024VCR-GauS: View Consistent Depth-Normal Regularizer for Gaussian Surface Reconstructioncited by 50
  2. 2024NeRFs in Robotics: A Surveycited by 16
  3. 2025FreeSplat++: Generalizable 3D Gaussian Splatting for Efficient Indoor Scene Reconstructioncited by 15
  4. 2025econSG: Efficient and Multi-view Consistent Open-Vocabulary 3D Semantic Gaussianscited by 3
  5. 2025RAZER: Robust Accelerated Zero-Shot 3D Open-Vocabulary Panoptic Reconstruction with Spatio-Temporal Aggregationcited by 2
  6. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  7. 2025UniFField: A Generalizable Unified Neural Feature Field for Visual, Semantic, and Spatial Uncertainties in Any Scenecited by 0
  8. 2026Robust Dreamer: Deviation-Aware Latent Gaussian Memory for Action-Controlled AR Video Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculum
  2. 2023Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representation
  3. 2025Unleashing the Multi-View Fusion Potential: Noise Correction in VLM for Open-Vocabulary 3D Scene Understanding
  4. 2025Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding
  5. 2024Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding
  6. 2024Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understanding
  7. 2023Weakly Supervised 3D Open-vocabulary Segmentation
  8. 2026GoDeep: Annotation-Free Open-Vocabulary 3D Scene Understanding via Language-Space Lifting