3D Scene Graph Guided Vision-Language Pre-training
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
3D vision-language (VL) reasoning has gained significant attention due to its potential to bridge the 3D physical world with natural language descriptions. Existing approaches typically follow task-specific, highly specialized paradigms. Therefore, these methods focus on a limited range of reasoning sub-tasks and rely heavily on the hand-crafted modules and auxiliary losses. This highlights the need for a simpler, unified and general-purpose model. In this paper, we leverage the inherent connection between 3D scene graphs and natural language, proposing a 3D scene graph-guided vision-language pre-training (VLP) framework. Our approach utilizes modality encoders, graph convolutional layers and cross-attention layers to learn universal representations that adapt to a variety of 3D VL reasoning tasks, thereby eliminating the need for task-specific designs. The pre-training objectives include: 1) Scene graph-guided contrastive learning, which leverages the strong correlation between 3D scene graphs and natural language to align 3D objects with textual features at various fine-grained levels; and 2) Masked modality learning, which uses cross-modality information to reconstruct masked words and 3D objects. Instead of directly reconstructing the 3D point clouds of masked objects, we use position clues to predict their semantic categories. Extensive experiments demonstrate that our pre-training model, when fine-tuned on several downstream tasks, achieves performance comparable to or better than existing methods in tasks such as 3D visual grounding, 3D dense captioning, and 3D question answering.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 62 references that are in the map. First posted on arXiv 27 Nov 2024.
Builds on
12 of 62 references.
Closest ancestors · 1
Same direction, newest first
- 2023Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes4 Jun
Other references · 11
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
- 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
- 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
- 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
- 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 78
- 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
- 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
- 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
- 2023Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph predictioncited by 19
- 2022Contextual Modeling for 3D Dense Captioning on Point Cloudscited by 18
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding
- 2025View-on-Graph: Zero-shot 3D Visual Grounding via Vision-Language Reasoning on Scene Graphs
- 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning
- 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding
- 20243D Vision and Language Pretraining with Large-Scale Synthetic Data
- 2026An overview of 3D Vision-Language Models
- 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding
- 20253D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation