mlsanityData to 14 Oct 2026

CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes

arXiv 2304.06061 ↗First posted 12 Apr 2023cs.CVcited by 55

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Training models to apply linguistic knowledge and visual concepts from 2D images to 3D world understanding is a promising direction that researchers have only recently started to explore. In this work, we design a novel 3D pre-training Vision-Language method that helps a model learn semantically meaningful and transferable 3D scene point cloud representations. We inject the representational power of the popular CLIP model into our 3D encoder by aligning the encoded 3D scene features with the corresponding 2D image and text embeddings produced by CLIP. To assess our model's 3D world reasoning capability, we evaluate it on the downstream task of 3D Visual Question Answering. Experimental quantitative and qualitative results show that our pre-training method outperforms state-of-the-art works in this task and leads to an interpretable representation of 3D scene features.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 31 references that are in the map. First posted on arXiv 12 Apr 2023.

Builds on

26 of 31 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  5. 2015VQA: Visual Question Answeringcited by 2,506
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  9. 2018Open3D: A Modern Library for 3D Data Processingcited by 721
  10. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  11. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  12. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 504
14 more
  1. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  2. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 408
  3. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  4. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  5. 2017Embodied Question Answeringcited by 336
  6. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  7. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  8. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  9. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  10. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  11. 2021SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representationscited by 55
  12. 2021Language Grounding with 3D Objectscited by 43
  13. 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
  14. 2021Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretrainingcited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 19; 2025: 23; 2026: 5.

55 papers.

Sort:

By area:

  1. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  2. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  3. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  4. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  5. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  6. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 81
  7. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 68
  8. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 54
  9. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 51
  10. 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversioncited by 43
  11. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  12. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
38 more
  1. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 25
  2. 2024Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Modelscited by 23
  3. 2024Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQAcited by 21
  4. 20243D Vision and Language Pretraining with Large-Scale Synthetic Datacited by 21
  5. 2024Context-based and Diversity-driven Specificity in Compositional Zero-Shot Learningcited by 20
  6. 2024Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Modelscited by 19
  7. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillationcited by 18
  8. 2024Open-Vocabulary Calibration for Fine-tuned CLIPcited by 15
  9. 2024What do we learn from inverting CLIP models?cited by 13
  10. 2024RAVE: Residual Vector Embedding for CLIP-Guided Backlit Image Enhancementcited by 11
  11. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answeringcited by 11
  12. 20243D Question Answering for City Scene Understandingcited by 10
  13. 2024External Knowledge Enhanced 3D Scene Generation from Sketchcited by 9
  14. 2024Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Modelscited by 9
  15. 2024PerLA: Perceptive 3D Language Assistantcited by 9
  16. 2025Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Modelscited by 9
  17. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
  18. 2025HIS-GPT: Towards 3D Human-In-Scene Multimodal Understandingcited by 6
  19. 2023Generating Context-Aware Natural Answers for Questions in 3D Scenescited by 5
  20. 2025CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoningcited by 5
  21. 2024Probabilistic Vision-Language Representation for Weakly Supervised Temporal Action Localizationcited by 3
  22. 2024OE3DIS: Open-Ended 3D Point Cloud Instance Segmentationcited by 3
  23. 2025City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learningcited by 3
  24. 2026Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRAcited by 3
  25. 2026IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignmentcited by 3
  26. 2025Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planningcited by 2
  27. 2025ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matchingcited by 2
  28. 2026Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLMcited by 2
  29. 2023Realistic Speech-to-Face Generation with Speech-Conditioned Latent Diffusion Model with Face Priorcited by 1
  30. 2025NeuroVoxel-LM: Language-Aligned 3D Perception via Dynamic Voxelization and Meta-Embeddingcited by 1
  31. 2025Text-Scene: A Scene-to-Language Parsing Framework for 3D Scene Understandingcited by 1
  32. 2025Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understandingcited by 1
  33. 2026Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spacescited by 1
  34. 2023Variational Information Pursuit with Large Language and Multimodal Models for Interpretable Predictionscited by 0
  35. 2025Semantically Guided Dynamic Visual Prototype Refinement for Compositional Zero-Shot Learningcited by 0
  36. 2025HCNQA: Enhancing 3D VQA with Hierarchical Concentration Narrowing Supervisioncited by 0
  37. 2025A Study of the Framework and Real-World Applications of Language Embedding for 3D Scene Understandingcited by 0
  38. 2025Improving Adversarial Robustness of Zero-Shot CLIP with Confidence-Aware Weightingcited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes
  2. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data
  3. 20233D Concept Learning and Reasoning from Multi-View Images
  4. 2025Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotations
  5. 2026Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation
  6. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP
  7. 20253D Question Answering via only 2D Vision-Language Models
  8. 2026PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding