mlsanityData to 14 Oct 2026

Towards Label-free Scene Understanding by Vision Foundation Models

arXiv 2306.03899 ↗First posted 6 Jun 2023cs.CVcited by 54

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision foundation models such as Contrastive Vision-Language Pre-training (CLIP) and Segment Anything (SAM) have demonstrated impressive zero-shot performance on image classification and segmentation tasks. However, the incorporation of CLIP and SAM for label-free scene understanding has yet to be explored. In this paper, we investigate the potential of vision foundation models in enabling networks to comprehend 2D and 3D worlds without labelled data. The primary challenge lies in effectively supervising networks under extremely noisy pseudo labels, which are generated by CLIP and further exacerbated during the propagation from the 2D to the 3D domain. To tackle these challenges, we propose a novel Cross-modality Noisy Supervision (CNS) method that leverages the strengths of CLIP and SAM to supervise 2D and 3D networks simultaneously. In particular, we introduce a prediction consistency regularization to co-train 2D and 3D networks, then further impose the networks' latent space consistency using the SAM's robust feature representation. Experiments conducted on diverse indoor and outdoor datasets demonstrate the superior performance of our method in understanding 2D and 3D open environments. Our 2D and 3D network achieves label-free semantic segmentation with 28.4% and 33.5% mIoU on ScanNet, improving 4.7% and 7.9%, respectively. For nuImages and nuScenes datasets, the performance is 22.1% and 26.8% with improvements of 3.5% and 6.0%, respectively. Code is available. (https://github.com/runnanchen/Label-Free-Scene-Understanding).

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 69 of its 94 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

69 of 94 references.

References · 69

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2023Segment Anythingcited by 8,234
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
38 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  4. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  5. 2021Vision Transformers for Dense Predictioncited by 1,803
  6. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  7. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  8. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  9. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  10. 2017A Closer Look at Memorization in Deep Networkscited by 942
  11. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  12. 2022Language-driven Semantic Segmentationcited by 717
  13. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 665
  14. 2021Extract Free Dense Labels from CLIPcited by 583
  15. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  16. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  17. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 463
  18. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 389
  19. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 388
  20. 2020Semi-Supervised Semantic Segmentation with Cross-Consistency Trainingcited by 376
  21. 2019Panoptic-DeepLabcited by 363
  22. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  23. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  24. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  25. 2019Zero-Shot Semantic Segmentationcited by 245
  26. 2022Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labelscited by 218
  27. 2019Are Anchor Points Really Indispensable in Label-Noise Learning?cited by 208
  28. 2021ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentationcited by 184
  29. 2020Hierarchical Multi-Scale Attention for Semantic Segmentationcited by 183
  30. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 181
  31. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  32. 2021RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentationcited by 173
  33. 2021Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Trackingcited by 166
  34. 2021(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Networkcited by 164
  35. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  36. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 158
  37. 2023Robo3D: Towards Robust and Reliable 3D Perception against Corruptionscited by 129
  38. 2023Rethinking Range View Representation for LiDAR Segmentationcited by 128

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 24; 2025: 20; 2026: 3.

54 papers.

Sort:

By area:

  1. 2023UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebasecited by 57
  2. 2024IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detectioncited by 57
  3. 2024Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Drivingcited by 54
  4. 2023FRNet: Frustum-Range Networks for Scalable LiDAR Segmentationcited by 50
  5. 2024Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Drivingcited by 41
  6. 2024OpenESS: Event-based Semantic Scene Understanding with Open Vocabulariescited by 38
  7. 2024On the Foundations of Earth and Climate Foundation Modelscited by 33
  8. 20244D Contrastive Superflows are Dense 3D Representation Learnerscited by 33
  9. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  10. 2024Is Your LiDAR Placement Optimized for 3D Scene Understanding?cited by 26
  11. 2024Multi-Space Alignments Towards Universal LiDAR Segmentationcited by 25
  12. 2025Embodied AI: From LLMs to World Modelscited by 24
38 more
  1. 2023See More and Know More: Zero-shot Point Cloud Segmentation via Multi-modal Visual Datacited by 21
  2. 2023Learning to Adapt SAM for Segmenting Cross-domain Point Cloudscited by 20
  3. 2025Perspective-Invariant 3D Object Detectioncited by 18
  4. 2024Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillationcited by 17
  5. 2024ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Imagescited by 17
  6. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  7. 2024Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3Dcited by 15
  8. 2023The RoboDepth Challenge: Methods and Advancements Towards Robust Depth Estimationcited by 13
  9. 2025Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understandingcited by 13
  10. 2024An Empirical Study of Training State-of-the-Art LiDAR Segmentation Modelscited by 12
  11. 20243D Weakly Supervised Semantic Segmentation with 2D Vision-Language Guidancecited by 12
  12. 2024Part2Object: Hierarchical Unsupervised 3D Instance Segmentationcited by 12
  13. 2024Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understandingcited by 11
  14. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenescited by 11
  15. 2024MixSup: Mixed-grained Supervision for Label-efficient LiDAR-based 3D Object Detectioncited by 9
  16. 2025Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priorscited by 9
  17. 2025Beyond One Shot, Beyond One Perspective: Cross-View and Long-Horizon Distillation for Better LiDAR Representationscited by 7
  18. 2025U4D: Uncertainty-Aware 4D World Modeling from LiDAR Sequencescited by 7
  19. 2024Mitigating Label Noise on Graph via Topological Sample Selectioncited by 6
  20. 2024AdaCo: Overcoming Visual Foundation Model Noise in 3D Semantic Segmentation via Adaptive Label Correctioncited by 6
  21. 20243D-AVS: LiDAR-based 3D Auto-Vocabulary Segmentationcited by 5
  22. 2025LOSC: LiDAR Open-voc Segmentation Consolidatorcited by 5
  23. 2024Vision Technologies with Applications in Traffic Surveillance Systems: A Holistic Surveycited by 4
  24. 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
  25. 2023Model2Scene: Learning 3D Scene Representation via Contrastive Language-CAD Models Pre-trainingcited by 3
  26. 2024Image-to-Lidar Relational Distillation for Autonomous Driving Datacited by 3
  27. 2024UniPLV: Towards Label-Efficient Open-World 3D Scene Understanding by Regional Visual Language Supervisioncited by 3
  28. 2024PanoSLAM: Panoptic 3D Scene Reconstruction via Gaussian SLAMcited by 3
  29. 2025City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learningcited by 3
  30. 2025ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updatescited by 3
  31. 2026AdaSFormer: Adaptive Serialized Transformers for Monocular Semantic Scene Completion from Indoor Environmentscited by 3
  32. 2024HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenescited by 2
  33. 2025Zero-Shot 3D Visual Grounding from Vision-Language Modelscited by 2
  34. 2025SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoningcited by 2
  35. 2023DORec: Decomposed Object Reconstruction and Segmentation Utilizing 2D Self-Supervised Featurescited by 1
  36. 2025GeoPurify: A Data-Efficient Geometric Distillation Framework for Open-Vocabulary 3D Segmentationcited by 1
  37. 2025ALISE: Annotation-Free LiDAR Instance Segmentation for Autonomous Drivingcited by 1
  38. 20253D Weakly Supervised Semantic Segmentation via Class-Aware and Geometry-Guided Pseudo-Label Refinementcited by 1

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP
  2. 2024Cross-Modal Self-Training: Aligning Images and Pointclouds to Learn Classification without Labels
  3. 2024Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understanding
  4. 2026SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes
  5. 2023Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Models
  6. 2025FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understanding
  7. 20243D Scene Graph Guided Vision-Language Pre-training
  8. 2026Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis