mlsanityData to 14 Oct 2026

Energy-Based Learning for Scene Graph Generation

arXiv 2103.02221 ↗First posted 3 Mar 2021cs.CV cs.LGcited by 104

Not on the map. Why?

Abstract (arXiv)

Traditional scene graph generation methods are trained using cross-entropy losses that treat objects and relationships as independent entities. Such a formulation, however, ignores the structure in the output space, in an inherently structured prediction problem. In this work, we introduce a novel energy-based learning framework for generating scene graphs. The proposed formulation allows for efficiently incorporating the structure of scene graphs in the output space. This additional constraint in the learning framework acts as an inductive bias and allows models to learn efficiently from a small number of labels. We use the proposed energy-based framework to train existing state-of-the-art models and obtain a significant performance improvement, of up to 21% and 27%, on the Visual Genome and GQA benchmark datasets, respectively. Furthermore, we showcase the learning efficiency of the proposed framework by demonstrating superior performance in the zero- and few-shot settings where data is scarce.

Key facts. Cited by 104 papers in the map (data through 14 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 3 Mar 2021.

Builds on

24 of 33 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  6. 2015Gated Graph Sequence Neural Networkscited by 796
  7. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  8. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 414
  9. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
  10. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 356
  11. 2016Visual Relationship Detection with Language Priorscited by 334
  12. 2018Image Generation from Scene Graphscited by 324
12 more
  1. 2018Graph R-CNN for Scene Graph Generationcited by 298
  2. 2018Bilinear Attention Networkscited by 280
  3. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 259
  4. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 255
  5. 2020GPS-Net: Graph Property Sensing Network for Scene Graph Generationcited by 166
  6. 2019Graphical Contrastive Losses for Scene Graph Parsingcited by 100
  7. 2020Bridging Knowledge Graphs to Generate Scene Graphscited by 86
  8. 2019Unpaired Image Captioning via Scene Graph Alignmentscited by 82
  9. 2018Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generationcited by 78
  10. 2019Learning Canonical Representations for Scene Graph to Image Generationcited by 60
  11. 2020Learning Visual Commonsense for Robust Scene Graph Generationcited by 47
  12. 2020Graph Density-Aware Losses for Novel Compositions in Scene Graph Generationcited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 33; 2023: 34; 2024: 16; 2025: 10; 2026: 3.

104 papers.

Sort:

By area:

  1. 2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
  2. 2022Panoptic Scene Graph Generationcited by 114
  3. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 103
  4. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 96
  5. 2022Fine-Grained Scene Graph Generation with Data Transfercited by 57
  6. 2022GFPose: Learning 3D Human Pose Prior with Gradient Fieldscited by 57
  7. 2022A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspectivecited by 56
  8. 2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
  9. 2023GenPose: Generative Category-level Object Pose Estimation via Diffusion Modelscited by 48
  10. 2024EGTR: Extracting Graph from Transformer for Scene Graph Generationcited by 47
  11. 2023Panoptic Video Scene Graph Generationcited by 46
  12. 2021Structured Sparse R-CNN for Direct Scene Graph Generationcited by 44
38 more
  1. 2023VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloudcited by 43
  2. 2023SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesiscited by 41
  3. 2023Prototype-based Embedding Network for Scene Graph Generationcited by 37
  4. 2023LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generationcited by 30
  5. 2023Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retentioncited by 29
  6. 2023SwinGNN: Rethinking Permutation Invariance in Diffusion Models for Graph Generationcited by 28
  7. 2022Fine-Grained Predicates Learning for Scene Graph Generationcited by 27
  8. 2021Segmentation-grounded Scene Graph Generationcited by 25
  9. 2022Iterative Scene Graph Generationcited by 24
  10. 2023Devil's on the Edges: Selective Quad Attention for Scene Graph Generationcited by 23
  11. 2023GraspGF: Learning Score-based Grasping Primitive for Human-assisting Dexterous Graspingcited by 23
  12. 2022Iterative Scene Graph Generation with Generative Transformerscited by 22
  13. 20244D Panoptic Scene Graph Generationcited by 22
  14. 2021Not All Relations are Equal: Mining Informative Labels for Scene Graph Generationcited by 21
  15. 2023Unbiased Scene Graph Generation via Two-stage Causal Modelingcited by 21
  16. 2023SGRec3D: Self-Supervised 3D Scene Graph Learning via Object-Level Scene Reconstructioncited by 21
  17. 2022Meta Spatio-Temporal Debiasing for Video Scene Graph Generationcited by 20
  18. 2023LAW-Diffusion: Complex Scene Generation by Diffusion with Layoutscited by 20
  19. 2023Vision Relation Transformer for Unbiased Scene Graph Generationcited by 18
  20. 2025MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphscited by 16
  21. 2021From SLAM to Situational Awareness: Challenges and Surveycited by 15
  22. 2022Hierarchical Memory Learning for Fine-Grained Scene Graph Generationcited by 15
  23. 2025Generative Models in Decision Making: A Surveycited by 15
  24. 2022Adaptive Fine-Grained Predicates Learning for Scene Graph Generationcited by 14
  25. 2023Manga109Dialog: A Large-scale Dialogue Dataset for Comics Speaker Detectioncited by 14
  26. 2024HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generationcited by 14
  27. 2025A Causal Adjustment Module for Debiasing Scene Graph Generationcited by 14
  28. 2023Score-PA: Score-based 3D Part Assemblycited by 12
  29. 2025HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptationcited by 12
  30. 2024Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detectioncited by 11
  31. 2025DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generationcited by 11
  32. 2022Dual-branch Hybrid Learning Network for Unbiased Scene Graph Generationcited by 10
  33. 2022Knowledge-augmented Deep Learning and Its Applications: A Surveycited by 10
  34. 2024Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generationcited by 10
  35. 2021RelTransformer: A Transformer-Based Long-Tail Visual Relationship Recognitioncited by 9
  36. 2022Unsupervised Vision-Language Parsing: Seamlessly Bridging Visual Scene Graphs with Language Structures via Dependency Relationshipscited by 9
  37. 2023Environment-Invariant Curriculum Relation Learning for Fine-Grained Scene Graph Generationcited by 9
  38. 2022Probabilistic Debiasing of Scene Graphscited by 8

50 most cited of 104 All 104 (JSON)