mlsanityData to 14 Oct 2026

Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding

arXiv 2305.10714 ↗First posted 18 May 2023cs.CVcited by 25

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

In recent years, vision language pre-training frameworks have made significant progress in natural language processing and computer vision, achieving remarkable performance improvement on various downstream tasks. However, when extended to point cloud data, existing works mainly focus on building task-specific models, and fail to extract universal 3D vision-language embedding that generalize well. We carefully investigate three common tasks in semantic 3D scene understanding, and derive key insights into the development of a pre-training model. Motivated by these observations, we propose a vision-language pre-training framework 3DVLP (3D vision-language pre-training with object contrastive learning), which transfers flexibly on 3D vision-language downstream tasks. 3DVLP takes visual grounding as the proxy task and introduces Object-level IoU-guided Detection (OID) loss to obtain high-quality proposals in the scene. Moreover, we design Object-level Cross-Contrastive alignment (OCC) task and Object-level Self-Contrastive learning (OSC) task to align the objects with descriptions and distinguish different objects in the scene, respectively. Extensive experiments verify the excellent performance of 3DVLP on three 3D vision-language tasks, reflecting its superiority in semantic 3D scene understanding.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 58 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

43 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
30 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  3. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,577
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  7. 2019When Does Label Smoothing Help?cited by 988
  8. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  9. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  10. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  11. 2019Deep Learning for 3D Point Clouds: A Surveycited by 558
  12. 2019Distance-IoU Loss: Faster and Better Learning for Bounding Box Regressioncited by 459
  13. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  14. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  15. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  16. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  17. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  18. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  19. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 160
  20. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
  21. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  22. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  23. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  24. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 110
  25. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  26. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  27. 2023Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Modelscited by 68
  28. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  29. 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
  30. 2022Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baselinecited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 14; 2026: 4.

25 papers.

Sort:

By area:

  1. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 352
  2. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  3. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 51
  4. 2023Towards Compact 3D Representations via Point Feature Enhancement Masked Autoencoderscited by 42
  5. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 25
  6. 2024Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQAcited by 21
  7. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  8. 20253D CoCa: Contrastive Learners are 3D Captionerscited by 13
  9. 2024LCM: Locally Constrained Compact Point Cloud Model for Masked Point Modelingcited by 12
  10. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answeringcited by 11
  11. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  12. 20253D Question Answering via only 2D Vision-Language Modelscited by 10
13 more
  1. 2025Advancing 3D Scene Understanding with MV-ScanQA Multi-View Reasoning Evaluation and TripAlign Pre-training Datasetcited by 8
  2. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 5
  3. 2025A Review of 3D Object Detection with Vision-Language Modelscited by 4
  4. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
  5. 20263D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligencecited by 3
  6. 2024PMT-MAE: Dual-Branch Self-Supervised Learning with Distillation for Efficient Point Cloud Classificationcited by 0
  7. 20253DWG: 3D Weakly Supervised Visual Grounding via Category and Instance-Level Alignmentcited by 0
  8. 2025LSVG: Language-Guided Scene Graphs with 2D-Assisted Multi-Modal Encoding for 3D Visual Groundingcited by 0
  9. 2025B2N3D: Progressive Learning from Binary to N-ary Relationships for 3D Object Groundingcited by 0
  10. 2025Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understandingcited by 0
  11. 2026Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportationcited by 0
  12. 20263D-Aware VLMs with Implicit and Explicit Geometriescited by 0
  13. 2026CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20243D Scene Graph Guided Vision-Language Pre-training
  2. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data
  3. 2026An overview of 3D Vision-Language Models
  4. 2023Generalized Robot 3D Vision-Language Model with Fast Rendering and Pre-Training Vision-Language Alignment
  5. 2023Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph prediction
  6. 2023Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes
  7. 20243D Vision and Language Pretraining with Large-Scale Synthetic Data
  8. 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding