mlsanityData to 7 Oct 2026

Learning to Compose Dynamic Tree Structures for Visual Contexts

arXiv 1812.01880 ↗First posted 5 Dec 2018cs.CVcited by 258

Not on the map. Why?

Abstract (arXiv)

We propose to compose dynamic tree structures that place the objects in an image into a visual context, helping visual reasoning tasks such as scene graph generation and visual Q&A. Our visual context tree model, dubbed VCTree, has two key advantages over existing structured object representations including chains and fully-connected graphs: 1) The efficient and expressive binary tree encodes the inherent parallel/hierarchical relationships among objects, e.g., "clothes" and "pants" are usually co-occur and belong to "person"; 2) the dynamic structure varies from image to image and task to task, allowing more content-/task-specific message passing among objects. To construct a VCTree, we design a score function that calculates the task-dependent validity between each object pair, and the tree is the binary version of the maximum spanning tree from the score matrix. Then, visual contexts are encoded by bidirectional TreeLSTM and decoded by task-specific models. We develop a hybrid learning procedure which integrates end-task supervised learning and the tree structure reinforcement learning, where the former's evaluation result serves as a self-critic for the latter's structure exploration. Experimental results on two benchmarks, which require reasoning over contexts: Visual Genome for scene graph generation and VQA2.0 for visual Q&A, show that VCTree outperforms state-of-the-art results while discovering interpretable visual context structures.

Key facts. Cited by 258 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 39; 2022: 47; 2023: 69; 2024: 51; 2025: 30; 2026: 22.

258 papers.

Sort:

By area:

  1. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  2. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  3. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  4. 2021Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generationcited by 137
  5. 2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
  6. 2022Panoptic Scene Graph Generationcited by 114
  7. 2021Energy-Based Learning for Scene Graph Generationcited by 104
  8. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 102
  9. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
  10. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 95
  11. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 95
  12. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
38 more
  1. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  2. 2022Scene Graph Generation: A Comprehensive Surveycited by 73
  3. 2021Recovering the Unbiased Scene Graphs from the Biased Onescited by 71
  4. 2022Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generationcited by 67
  5. 2021Learning of Visual Relations: The Devil is in the Tailscited by 65
  6. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  7. 2022The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generationcited by 60
  8. 2021Fully Convolutional Scene Graph Generationcited by 57
  9. 2022Fine-Grained Scene Graph Generation with Data Transfercited by 57
  10. 2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
  11. 2021Deep Dual Consecutive Network for Human Pose Estimationcited by 51
  12. 2021Target Adaptive Context Aggregation for Video Scene Graph Generationcited by 51
  13. 2021From General to Specific: Informative Scene Graph Generation via Balance Adjustmentcited by 51
  14. 2023Action Scene Graphs for Long-Form Understanding of Egocentric Videoscited by 51
  15. 2023Panoptic Video Scene Graph Generationcited by 46
  16. 2024EGTR: Extracting Graph from Transformer for Scene Graph Generationcited by 45
  17. 2021Structured Sparse R-CNN for Direct Scene Graph Generationcited by 44
  18. 2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 44
  19. 2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
  20. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Modelscited by 43
  21. 2023VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloudcited by 42
  22. 2023Panoptic Scene Graph Generation with Semantics-Prototype Learningcited by 42
  23. 20213D Question Answeringcited by 41
  24. 2023Unbiased Scene Graph Generation in Videoscited by 38
  25. 2023Prototype-based Embedding Network for Scene Graph Generationcited by 37
  26. 2023Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representationscited by 37
  27. 2022MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answeringcited by 36
  28. 2021Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generationcited by 35
  29. 2023Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open Worldcited by 35
  30. 2021In Defense of Scene Graphs for Image Captioningcited by 31
  31. 2024STAR: A First-Ever Dataset and A Large-Scale Benchmark for Scene Graph Generation in Large-Size Satellite Imagerycited by 31
  32. 2021Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphscited by 30
  33. 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Modelscited by 30
  34. 2023Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retentioncited by 29
  35. 2022RU-Net: Regularized Unrolling Network for Scene Graph Generationcited by 28
  36. 2022Fine-Grained Predicates Learning for Scene Graph Generationcited by 27
  37. 2021Visual Distant Supervision for Scene Graph Generationcited by 25
  38. 2021Segmentation-grounded Scene Graph Generationcited by 25

50 most cited of 258 All 258 (JSON)