mlsanityData to 7 Oct 2026

3D Vision and Language Pretraining with Large-Scale Synthetic Data

arXiv 2407.06084 ↗First posted 8 Jul 2024cs.CVcited by 21

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

3D Vision-Language Pre-training (3D-VLP) aims to provide a pre-train model which can bridge 3D scenes with natural language, which is an important technique for embodied intelligence. However, current 3D-VLP datasets are hindered by limited scene-level diversity and insufficient fine-grained annotations (only 1.2K scenes and 280K textual annotations in ScanScribe), primarily due to the labor-intensive of collecting and annotating 3D scenes. To overcome these obstacles, we construct SynVL3D, a comprehensive synthetic scene-text corpus with 10K indoor scenes and 1M descriptions at object, view, and room levels, which has the advantages of diverse scene data, rich textual descriptions, multi-grained 3D-text associations, and low collection cost. Utilizing the rich annotations in SynVL3D, we pre-train a simple and unified Transformer for aligning 3D and language with multi-grained pretraining tasks. Moreover, we propose a synthetic-to-real domain adaptation in downstream task fine-tuning process to address the domain shift. Through extensive experiments, we verify the effectiveness of our model design by achieving state-of-the-art performance on downstream tasks including visual grounding, dense captioning, and question answering.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 45 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

32 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes4 Jun
  2. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes12 Apr

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  7. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  8. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  9. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  10. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  11. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  12. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
18 more
  1. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  2. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
  3. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  4. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  5. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  6. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  7. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  8. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  9. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  10. 2020ePointDA: An End-to-End Simulation-to-Real Domain Adaptation Framework for LiDAR Point Cloud Segmentationcited by 56
  11. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  12. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  13. 2022ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenescited by 32
  14. 2024Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQAcited by 21
  15. 2022Toward Explainable and Fine-Grained 3D Grounding through Referring Textual Phrasescited by 16
  16. 2022DODA: Data-oriented Sim-to-Real Domain Adaptation for 3D Semantic Segmentationcited by 13
  17. 2022Uncertainty-Induced Transferability Representation for Source-Free Unsupervised Domain Adaptationcited by 13
  18. 2022Delving into the Continuous Domain Adaptationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 14; 2026: 5.

21 papers.

Sort:

By area:

  1. 2024Training-free Video Temporal Grounding using Large-scale Pre-trained Modelscited by 28
  2. 2025GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyondcited by 12
  3. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  4. 20253D Question Answering via only 2D Vision-Language Modelscited by 10
  5. 2025Advancing 3D Scene Understanding with MV-ScanQA Multi-View Reasoning Evaluation and TripAlign Pre-training Datasetcited by 8
  6. 2025Chorus: Multi-Teacher Pretraining for Holistic 3D Gaussian Scene Encodingcited by 8
  7. 2025Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibrationcited by 7
  8. 2024ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Groundingcited by 6
  9. 2025Integrating Chain-of-Thought for Multimodal Alignment: A Study on 3D Vision-Language Learningcited by 5
  10. 2025AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoningcited by 5
  11. 2026OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Groundingcited by 4
  12. 2025TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferringcited by 3
9 more
  1. 2025InteractMove: Text-Controlled Human-Object Interaction Generation in 3D Scenes with Movable Objectscited by 3
  2. 2025Investigating Domain Gaps for Indoor 3D Object Detectioncited by 2
  3. 2026SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentationcited by 2
  4. 2026Scalable Object Relation Encoding for Better 3D Spatial Reasoning in Large Language Modelscited by 1
  5. 2025Prototype-Aware Multimodal Alignment for Open-Vocabulary Visual Groundingcited by 0
  6. 2025SweeperBot: Making 3D Browsing Accessible through View Analysis and Visual Question Answeringcited by 0
  7. 2025Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understandingcited by 0
  8. 2026Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportationcited by 0
  9. 2026Scaling Diverse Language Generation for 3D Visual Groundingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20243D Scene Graph Guided Vision-Language Pre-training
  2. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning
  3. 2024VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation
  4. 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding
  5. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  6. 2025Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs
  7. 2026VLM3: Vision Language Models Are Native 3D Learners
  8. 2025Vision-Language Models as Differentiable Semantic and Spatial Rewards for Text-to-3D Generation