mlsanityData to 13 Oct 2026

Generating Visual Spatial Description via Holistic 3D Scene Understanding

arXiv 2305.11768 ↗First posted 19 May 2023cs.CV cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Visual spatial description (VSD) aims to generate texts that describe the spatial relations of the given objects within images. Existing VSD work merely models the 2D geometrical vision features, thus inevitably falling prey to the problem of skewed spatial understanding of target objects. In this work, we investigate the incorporation of 3D scene features for VSD. With an external 3D scene extractor, we obtain the 3D objects and scene features for input images, based on which we construct a target object-centered 3D spatial scene graph (Go3D-S2G), such that we model the spatial semantics of target objects within the holistic 3D scenes. Besides, we propose a scene subgraph selecting mechanism, sampling topologically-diverse subgraphs from Go3D-S2G, where the diverse local structure features are navigated to yield spatially-diversified text generation. Experimental results on two VSD datasets demonstrate that our framework outperforms the baselines significantly, especially improving on the cases with complex visual spatial relations. Meanwhile, our method can produce more spatially-diversified generation. Code is available at https://github.com/zhaoyucs/VSD.

Key facts. Cited by 16 papers in the map (data through 13 Oct 2026). Builds on 36 of its 59 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

36 of 59 references.

References · 36

  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,625
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
  3. 2015VQA: Visual Question Answeringcited by 2,495
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,220
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  6. 2014Show and Tell: A Neural Image Caption Generatorcited by 878
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 744
  8. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  9. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  10. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  11. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 227
  12. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 172
24 more
  1. 2017Encoding Sentences with Graph Convolutional Networks for Semantic Role Labelingcited by 137
  2. 2021Holistic 3D Scene Understanding from a Single Image with Implicit Representationcited by 94
  3. 2021Unified Named Entity Recognition as Word-Word Relation Classificationcited by 90
  4. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 79
  5. 2018Holistic 3D Scene Parsing and Reconstruction from a Single RGB Imagecited by 69
  6. 2018Show, Control and Tell: A Framework for Generating Controllable and Grounded Captionscited by 65
  7. 2018Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speechcited by 55
  8. 2019SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognitioncited by 55
  9. 2018Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimationcited by 51
  10. 2020Cross-Lingual Semantic Role Labeling with High-Quality Translated Training Corpuscited by 49
  11. 2023LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Modelcited by 43
  12. 2021Human-like Controllable Image Captioning with Verb-specific Semantic Rolescited by 41
  13. 2021Learn from Syntax: Improving Pair-wise Aspect and Opinion Terms Extractionwith Rich Syntactic Knowledgecited by 31
  14. 2018SemStyle: Learning to Generate Stylised Image Captions using Unaligned Textcited by 27
  15. 2021Mastering the Explicit Opinion-role Interaction: Syntax-aided Neural Transition System for Unified Opinion Role Labelingcited by 25
  16. 2020Retrofitting Structure-aware Transformer Language Model for End Taskscited by 24
  17. 2017Representation Learning for Grounded Spatial Reasoningcited by 23
  18. 2021End-to-end Semantic Role Labeling with Neural Transition-based Modelcited by 22
  19. 2022Effective Token Graph Modeling using a Novel Labeling Strategy for Structured Sentiment Analysiscited by 22
  20. 2023On the Robustness of Aspect-based Sentiment Analysis: Rethinking Model, Data, and Trainingcited by 20
  21. 2022Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Expertscited by 19
  22. 2022Conversational Semantic Role Labeling with Predicate-Oriented Latent Graphcited by 14
  23. 2022Visual Spatial Description: Controlled Spatial-Oriented Image-to-Text Generationcited by 6
  24. 2017RUR53: an Unmanned Ground Vehicle for Navigation, Recognition and Manipulationcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 10; 2024: 5; 2025: 1.

16 papers.

Sort:

By area:

  1. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 152
  2. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generationcited by 92
  3. 2023Panoptic Scene Graph Generation with Semantics-Prototype Learningcited by 42
  4. 20233D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentationcited by 36
  5. 2023Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labelingcited by 15
  6. 2023Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognitioncited by 13
  7. 2023Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representationcited by 12
  8. 2023Domain-wise Invariant Learning for Panoptic Scene Graph Generationcited by 7
  9. 2024LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Descriptioncited by 7
  10. 2023A Bi-directional Multi-hop Inference Model for Joint Dialog Sentiment Classification and Act Recognitioncited by 2
  11. 20253DResT: A Strong Baseline for Semi-Supervised 3D Referring Expression Segmentationcited by 2
  12. 2023DialogRE^C+: An Extension of DialogRE to Investigate How Much Coreference Helps Relation Extraction in Dialogscited by 1
4 more
  1. 2023JM3D & JM3D-LLM: Elevating 3D Understanding with Joint Multi-modal Cuescited by 1
  2. 2024Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-Imagecited by 1
  3. 2024Modeling Unified Semantic Discourse Structure for High-quality Headline Generationcited by 0
  4. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 0