mlsanityData to 14 Oct 2026

VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud

arXiv 2303.14408 ↗First posted 25 Mar 2023cs.CVcited by 43

Not on the map. Why?

Abstract (arXiv)

The task of 3D semantic scene graph (3DSSG) prediction in the point cloud is challenging since (1) the 3D point cloud only captures geometric structures with limited semantics compared to 2D images, and (2) long-tailed relation distribution inherently hinders the learning of unbiased prediction. Since 2D images provide rich semantics and scene graphs are in nature coped with languages, in this study, we propose Visual-Linguistic Semantics Assisted Training (VL-SAT) scheme that can significantly empower 3DSSG prediction models with discrimination about long-tailed and ambiguous semantic relations. The key idea is to train a powerful multi-modal oracle model to assist the 3D model. This oracle learns reliable structural representations based on semantics from vision, language, and 3D geometry, and its benefits can be heterogeneously passed to the 3D model during the training stage. By effectively utilizing visual-linguistic semantics in training, our VL-SAT can significantly boost common 3DSSG prediction models, such as SGFN and SGGpoint, only with 3D inputs in the inference stage, especially when dealing with tail relation triplets. Comprehensive evaluations and ablation studies on the 3DSSG dataset have validated the effectiveness of the proposed scheme. Code is available at https://github.com/wz7in/CVPR2023-VLSAT.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 48 of its 56 references that are in the map. First posted on arXiv 25 Mar 2023.

Builds on

48 of 56 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  8. 2017Decoupled Weight Decay Regularizationcited by 1,831
  9. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  10. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 941
  11. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 891
  12. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
36 more
  1. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  2. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 466
  3. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  4. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  5. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  6. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 415
  7. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 414
  8. 2019From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Networkcited by 368
  9. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 356
  10. 2016Visual Relationship Detection with Language Priorscited by 334
  11. 2018Graph R-CNN for Scene Graph Generationcited by 298
  12. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  13. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 259
  14. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 208
  15. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 188
  16. 2017PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimationcited by 170
  17. 2021Multimodal Virtual Point 3D Detectioncited by 166
  18. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 164
  19. 2019MVX-Net: Multimodal VoxelNet for 3D Object Detectioncited by 163
  20. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 163
  21. 2020CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detectioncited by 147
  22. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  23. 2020H3DNet: 3D Object Detection Using Hybrid Geometric Primitivescited by 123
  24. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 120
  25. 2020ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votescited by 119
  26. 2021Energy-Based Learning for Scene Graph Generationcited by 104
  27. 2020Bridging Knowledge Graphs to Generate Scene Graphscited by 86
  28. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  29. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  30. 2021Back-tracing Representative Points for Voting-based 3D Object Detection in Point Cloudscited by 67
  31. 2021Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysiscited by 49
  32. 2018LinkNet: Relational Embedding for Scene Graphcited by 45
  33. 2020Classification by Attention: Scene Graph Classification with Prior Knowledgecited by 26
  34. 2020Exploring Long Tail Visual Relationship Recognition with Large Vocabularycited by 16
  35. 2017Natural Language Guided Visual Relationship Detectioncited by 15
  36. 2022HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectorscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 9; 2025: 13; 2026: 18.

43 papers.

Sort:

By area:

  1. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 80
  2. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 54
  3. 2024DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Modelscited by 33
  4. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  5. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 27
  6. 2025FunGraph: Functionality Aware 3D Scene Graphs for Language-Prompted Scene Interactioncited by 23
  7. 2023SGRec3D: Self-Supervised 3D Scene Graph Learning via Object-Level Scene Reconstructioncited by 21
  8. 2023A Survey of Label-Efficient Deep Learning for 3D Point Cloudscited by 20
  9. 2023Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph predictioncited by 19
  10. 2026Multiplayer Interactive World Models with Representation Autoencoderscited by 19
  11. 2024RelationField: Relate Anything in Radiance Fieldscited by 16
  12. 2025Universal Scene Graph Generationcited by 11
31 more
  1. 2025FROSS: Faster-than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Imagescited by 10
  2. 2025KeySG: Hierarchical Keyframe-Based 3D Scene Graphscited by 10
  3. 2024Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Roomscited by 6
  4. 2025Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scenecited by 6
  5. 2026ReLaGS: Relational Language Gaussian Splattingcited by 6
  6. 2024Weakly-Supervised 3D Scene Graph Generation via Visual-Linguistic Assisted Pseudo-labelingcited by 5
  7. 2025Statistical Confidence Rescoring for Robust 3D Scene Graph Generation from Multi-View Imagescited by 5
  8. 2025Social 3D Scene Graphs: Modeling Human Actions and Relations for Interactive Service Robotscited by 5
  9. 2025Aion: Towards Hierarchical 4D Scene Graphs with Temporal Flow Dynamicscited by 5
  10. 2024Belief Scene Graphs: Expanding Partial Scenes with Objects through Computation of Expectationcited by 4
  11. 2025Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Predictioncited by 4
  12. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  13. 2025Estimating Commonsense Scene Composition on Belief Scene Graphscited by 3
  14. 2025FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understandingcited by 2
  15. 2025Edge-Centric Relational Reasoning for 3D Scene Graph Predictioncited by 2
  16. 2024S^2Former-OR: Single-Stage Bi-Modal Transformer for Scene Graph Generation in ORcited by 1
  17. 2026Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spacescited by 1
  18. 2026DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priorscited by 1
  19. 2026R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Videocited by 1
  20. 2026MA3DSG: Multi-Agent 3D Scene Graph Generation for Large-Scale Indoor Environmentscited by 0
  21. 2026Knowledge-Guided Manipulation Using Multi-Task Reinforcement Learningcited by 0
  22. 2026SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matchingcited by 0
  23. 2026CAPruner: Conceptual-Adjacent Scene Graph Pruner for Enhancing 3D Spatial Reasoning of Large Language Modelscited by 0
  24. 2026SGFormer++: Semantic Graph Transformer for Incremental 3D Scene Graph Generationcited by 0
  25. 2026Scaling Diverse Language Generation for 3D Visual Groundingcited by 0
  26. 2026From Scene-Centric to Observer-Centric: Modeling Observer-Aware Relations for 3D Scene Graph Generationcited by 0
  27. 2026NoPA: Non-Parametric Online 3D Scene Graph Generationcited by 0
  28. 2026CAIRN: Cross-Room 3D Scene Understanding with Topology-Aware Large Multimodal Modelscited by 0
  29. 2026PUF: Plug-and-Play Uncertainty-Aware Fusion for Online 3D Scene Graph Generationcited by 0
  30. 2026GraphWrit3R: End-to-End 3D Scene Graph Writingcited by 0
  31. 2026GHOST: Geometry-Constrained Hybrid Diffusion for Open-Vocabulary 3D Scene Graph Generationcited by 0