mlsanityData to 7 Oct 2026

RelWitness: Open-Vocabulary 3D Scene Graph Generation with Visual-Geometric Relation Witnesses

arXiv 2605.20823 ↗First posted 20 May 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Open-vocabulary 3D scene graph generation seeks to describe object instances and their relations with flexible natural-language predicates. The central difficulty is not only vocabulary expansion, but supervision reliability: relation annotations in 3D scene graph datasets are selective, and many valid object-pair relations are unannotated. We propose RelWitness, a framework for open-vocabulary 3D scene graph generation from posed RGB-D sequences under incomplete relation supervision. The key concept is a relation witness: a concrete visual-geometric cue that makes a relation observable in the captured scene. Support relations require contact and vertical ordering; containment requires enclosure; proximity requires metric closeness; orientation requires facing direction; and stable relations should persist across views where both objects are visible. RelWitness constructs relation witness records from RGB views, depth maps, reconstructed 3D geometry, role-sensitive text, object-prior null views, and multi-view consistency. A visual-geometric witness verifier assigns unannotated relation candidates to verified missing positives, reliable negatives, or uncertain unlabeled cases. A witness-guided positive-unlabeled objective then learns from incomplete annotations without turning every missing label into a negative. We further introduce witness-consistent decoding and an RGB-D missing-relation audit protocol. Simulated manuscript-planning experiments on 3DSSG/3RScan and ScanNet-derived open-vocabulary splits show the intended behavior: improved unseen-relation recognition, higher witness precision, lower hallucination, and reduced redundant relation phrases. All numerical results are planning values and must be replaced by reproduced measurements before submission

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 52 of its 68 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

52 of 68 references.

References · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2017Mask R-CNNcited by 7,259
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  9. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  12. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  3. 2020Point Transformercited by 1,462
  4. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  5. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  6. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  7. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  8. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  9. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  10. 2016Visual Relationship Detection with Language Priorscited by 333
  11. 2018Graph R-CNN for Scene Graph Generationcited by 296
  12. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  13. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  14. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 258
  15. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
  16. 2019Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mappingcited by 181
  17. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 163
  18. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 159
  19. 2021Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generationcited by 137
  20. 2022Panoptic Scene Graph Generationcited by 114
  21. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 102
  22. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  23. 2021Recovering the Unbiased Scene Graphs from the Biased Onescited by 71
  24. 2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
  25. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  26. 2019Scene Graph Prediction with Limited Labelscited by 32
  27. 2022Fine-Grained Predicates Learning for Scene Graph Generationcited by 27
  28. 2020Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generationcited by 22
  29. 2021Exploiting Scene Graphs for Human-Object Interaction Detectioncited by 22
  30. 2025Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognitioncited by 16
  31. 2023Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detectioncited by 15
  32. 2024OpenPSG: Open-set Panoptic Scene Graph Generation via Large Multimodal Modelscited by 15
  33. 2021Semantic Compositional Learning for Low-shot Scene Graph Generationcited by 12
  34. 2024MuAP: Multi-step Adaptive Prompt Learning for Vision-Language Model with Missing Modalitycited by 10
  35. 2025SPADE: Spatial-Aware Denoising Network for Open-vocabulary Panoptic Scene Graph Generation with Long- and Local-range Context Reasoningcited by 9
  36. 2017Deep Discrete Hashing with Self-supervised Pairwise Labelscited by 8
  37. 2021Semi-supervised Network Embedding with Differentiable Deep Quantisationcited by 7
  38. 2026Unbiased Dynamic Multimodal Fusioncited by 7

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.