mlsanityData to 14 Oct 2026

Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes

arXiv 2306.02329 ↗First posted 4 Jun 2023cs.CVcited by 22

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Training models to apply common-sense linguistic knowledge and visual concepts from 2D images to 3D scene understanding is a promising direction that researchers have only recently started to explore. However, it still remains understudied whether 2D distilled knowledge can provide useful representations for downstream 3D vision-language tasks such as 3D question answering. In this paper, we propose a novel 3D pre-training Vision-Language method, namely Multi-CLIP, that enables a model to learn language-grounded and transferable 3D scene point cloud representations. We leverage the representational power of the CLIP model by maximizing the agreement between the encoded 3D scene features and the corresponding 2D multi-view image and text embeddings in the CLIP space via a contrastive objective. To validate our approach, we consider the challenging downstream tasks of 3D Visual Question Answering (3D-VQA) and 3D Situated Question Answering (3D-SQA). To this end, we develop novel multi-modal transformer-based architectures and we demonstrate how our pre-training method can benefit their performance. Quantitative and qualitative experimental results show that Multi-CLIP outperforms state-of-the-art works across the downstream tasks of 3D-VQA and 3D-SQA and leads to a well-structured 3D scene feature space.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 37 references that are in the map. First posted on arXiv 4 Jun 2023.

Builds on

30 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2018Open3D: A Modern Library for 3D Data Processingcited by 721
17 more
  1. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  2. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  3. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 504
  4. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  5. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  6. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
  7. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  8. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  9. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  10. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  11. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  12. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  13. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  14. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  15. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  16. 2021Language Grounding with 3D Objectscited by 43
  17. 2021Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretrainingcited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 11; 2025: 7; 2026: 2.

22 papers.

Sort:

By area:

  1. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  2. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  3. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 54
  4. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  5. 2023Frozen Transformers in Language Models Are Effective Visual Encoder Layerscited by 39
  6. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 33
  7. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 27
  8. 2025ReasonGrounder: LVLM-Guided Hierarchical Feature Splatting for Open-Vocabulary 3D Visual Grounding and Reasoningcited by 25
  9. 2024Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQAcited by 21
  10. 20243D Vision and Language Pretraining with Large-Scale Synthetic Datacited by 21
  11. 2024Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammographycited by 9
  12. 2024PerLA: Perceptive 3D Language Assistantcited by 9
10 more
  1. 2025DSPNet: Dual-vision Scene Perception for Robust 3D Question Answeringcited by 9
  2. 2024OE3DIS: Open-Ended 3D Point Cloud Instance Segmentationcited by 3
  3. 2025Q2E: Query-to-Event Decomposition for Zero-Shot Multilingual Text-to-Video Retrievalcited by 3
  4. 2024Beyond Viewpoint: Robust 3D Object Recognition under Arbitrary Views through Joint Multi-Part Representationcited by 1
  5. 2025CXR-CML: Improved zero-shot classification of long-tailed multi-label diseases in Chest X-Rayscited by 1
  6. 2026Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spacescited by 1
  7. 2026Ego-Human Motion Prediction with 3D-Aware LLMcited by 1
  8. 20243D Scene Graph Guided Vision-Language Pre-trainingcited by 0
  9. 2025HCNQA: Enhancing 3D VQA with Hierarchical Concentration Narrowing Supervisioncited by 0
  10. 2025HMR3D: Hierarchical Multimodal Representation for 3D Scene Understanding with Large Vision-Language Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
  2. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data
  3. 20233D Concept Learning and Reasoning from Multi-View Images
  4. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  5. 2026Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation
  6. 20253D Question Answering via only 2D Vision-Language Models
  7. 20253D CoCa: Contrastive Learners are 3D Captioners
  8. 2026PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding