# Weakly-Supervised 3D Scene Graph Generation via Visual-Linguistic Assisted Pseudo-labeling arXiv: https://arxiv.org/abs/2404.02527 ยท HTML: https://mlsanity.com/paper/2404.02527/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 45 of its 68 references that are in the map. - First posted on arXiv 3 Apr 2024. ## Builds on (most cited first, 30 of 45) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014) - [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016) - [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017) - [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016) - [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015) - [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021) - [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018) - [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019) - [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019) - [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021) - [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017) - [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019) - [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021) - [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019) - [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021) - [Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions](https://mlsanity.com/paper/2004.03967/) (2020) - [RIO: 3D Object Instance Re-Localization in Changing Indoor Environments](https://mlsanity.com/paper/1908.06109/) (2019) - [Weakly Supervised Deep Detection Networks](https://mlsanity.com/paper/1511.02853/) (2015) - [Quaternion Knowledge Graph Embeddings](https://mlsanity.com/paper/1904.10281/) (2019) - [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019) - [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021) - [H3DNet: 3D Object Detection Using Hybrid Geometric Primitives](https://mlsanity.com/paper/2006.05682/) (2020) - [GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection](https://mlsanity.com/paper/2203.13954/) (2022) - [TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding](https://mlsanity.com/paper/2108.02388/) (2021) - [Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement](https://mlsanity.com/paper/2304.01195/) (2023) - [Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds](https://mlsanity.com/paper/2104.06114/) (2021) - [Learning to Generate Scene Graph from Natural Language Supervision](https://mlsanity.com/paper/2109.02227/) (2021) - [Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysis](https://mlsanity.com/paper/2103.05558/) (2021) ## Built on by (most cited first, 5 of 5) - [Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scene](https://mlsanity.com/paper/2503.15019/) (2025) - [Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction](https://mlsanity.com/paper/2510.04714/) (2025) - [3D Scene Graphs: Open Challenges and Future Directions](https://mlsanity.com/paper/2606.19383/) (2026) - [M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments](https://mlsanity.com/paper/2508.17044/) (2025) - [B2N3D: Progressive Learning from Binary to N-ary Relationships for 3D Object Grounding](https://mlsanity.com/paper/2510.10194/) (2025) ## Abstract (arXiv) Learning to build 3D scene graphs is essential for real-world perception in a structured and rich fashion. However, previous 3D scene graph generation methods utilize a fully supervised learning manner and require a large amount of entity-level annotation data of objects and relations, which is extremely resource-consuming and tedious to obtain. To tackle this problem, we propose 3D-VLAP, a weakly-supervised 3D scene graph generation method via Visual-Linguistic Assisted Pseudo-labeling. Specifically, our 3D-VLAP exploits the superior ability of current large-scale visual-linguistic models to align the semantics between texts and 2D images, as well as the naturally existing correspondences between 2D images and 3D point clouds, and thus implicitly constructs correspondences between texts and 3D point clouds. First, we establish the positional correspondence from 3D point clouds to 2D images via camera intrinsic and extrinsic parameters, thereby achieving alignment of 3D point clouds and 2D images. Subsequently, a large-scale cross-modal visual-linguistic model is employed to indirectly align 3D instances with the textual category labels of objects by matching 2D images with object category labels. The pseudo labels for objects and relations are then produced for 3D-VLAP model training by calculating the similarity between visual embeddings and textual category embeddings of objects and relations encoded by the visual-linguistic model, respectively. Ultimately, we design an edge self-attention based graph neural network to generate scene graphs of 3D point cloud scenes. Extensive experiments demonstrate that our 3D-VLAP achieves comparable results with current advanced fully supervised methods, meanwhile significantly alleviating the pressure of data annotation.