mlsanityData to 7 Oct 2026

Learning to Compose Visual Relations

arXiv 2111.09297 ↗First posted 17 Nov 2021cs.CV cs.AI cs.LGcited by 72

Not on the map. Why?

Cited by 25 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The visual world around us can be described as a structured set of objects and their associated relations. An image of a room may be conjured given only the description of the underlying objects and their associated relations. While there has been significant work on designing deep neural networks which may compose individual objects together, less work has been done on composing the individual relations between objects. A principal difficulty is that while the placement of objects is mutually independent, their relations are entangled and dependent on each other. To circumvent this issue, existing works primarily compose relations by utilizing a holistic encoder, in the form of text or graphs. In this work, we instead propose to represent each relation as an unnormalized density (an energy-based model), enabling us to compose separate relations in a factorized manner. We show that such a factorized decomposition allows the model to both generate and edit scenes that have multiple sets of relations more faithfully. We further show that decomposition enables our model to effectively understand the underlying relational scene structure. Project page at: https://composevisualrelations.github.io/.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 17 Nov 2021.

Builds on

41 of 50 references.

References · 41

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  7. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  8. 2016Generative Adversarial Text to Image Synthesiscited by 735
  9. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  10. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  11. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  12. 2018MolGAN: An implicit generative model for small molecular graphscited by 392
29 more
  1. 2018Image Generation from Scene Graphscited by 324
  2. 2020Integrated Task and Motion Planningcited by 281
  3. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  4. 2015DRAW: A Recurrent Neural Network For Image Generationcited by 190
  5. 2016A Theory of Generative ConvNetcited by 174
  6. 2020iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenescited by 151
  7. 2015Generating Images from Captions with Attentioncited by 135
  8. 2019LayoutGAN: Generating Graphic Layouts with Wireframe Discriminatorscited by 130
  9. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 127
  10. 2016Learning What and Where to Drawcited by 105
  11. 2019On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Modelscited by 105
  12. 2019Scene Graph Generation with External Knowledge and Image Reconstructioncited by 102
  13. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  14. 2020House-GAN: Relational Generative Adversarial Networks for Graph-constrained House Layout Generationcited by 86
  15. 2019Specifying Object Attributes and Relations in Interactive Scene Generationcited by 76
  16. 2019Flow Contrastive Estimation of Energy-Based Modelscited by 75
  17. 2017Parallel Multiscale Autoregressive Density Estimationcited by 71
  18. 2016Learning Physical Intuition of Block Towers by Examplecited by 66
  19. 2019Learning Canonical Representations for Scene Graph to Image Generationcited by 60
  20. 2019PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graphcited by 44
  21. 2016Deep Directed Generative Models with Energy-Based Probability Estimationcited by 34
  22. 2020Compositional Video Synthesis with Action Graphscited by 34
  23. 2019Exponential Family Estimation via Adversarial Dynamics Embeddingcited by 30
  24. 2020Compositional Visual Generation and Inference with Energy Based Modelscited by 25
  25. 2019Interactive Image Generation Using Scene Graphscited by 17
  26. 2019Using Scene Graph Context to Improve Image Generationcited by 12
  27. 2017Discovering objects and their relations from entangled scene representationscited by 11
  28. 2018Generative Modeling by Inclusive Neural Random Fields with Applications in Image Generation and Anomaly Detectioncited by 9
  29. 2021Exploiting Relationship for Complex-scene Image Generationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 14; 2023: 21; 2024: 18; 2025: 14; 2026: 4.

72 papers.

Sort:

By area:

  1. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  2. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  3. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  4. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 216
  5. 2023Teaching CLIP to Count to Tencited by 194
  6. 2023Video Language Planningcited by 160
  7. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  8. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  9. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  10. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  11. 2024Multi-LoRA Composition for Image Generationcited by 80
  12. 2024ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficultycited by 60
38 more
  1. 2023Concept Algebra for (Score-Based) Text-Controlled Generative Modelscited by 59
  2. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  3. 2023Visual Anagrams: Generating Multi-View Optical Illusions with Diffusion Modelscited by 46
  4. 2024TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generationcited by 42
  5. 2023Compositional Diffusion-Based Continuous Constraint Solverscited by 39
  6. 2024Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directionscited by 39
  7. 2023Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Modelscited by 36
  8. 2022VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programmingcited by 31
  9. 2023Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangementcited by 30
  10. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?cited by 30
  11. 2023Probabilistic Adaptation of Text-to-Video Modelscited by 29
  12. 2022Composing Ensembles of Pre-trained Models via Iterative Consensuscited by 28
  13. 2022Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Modelscited by 27
  14. 2023Compositional Sculpting of Iterative Generative Processescited by 26
  15. 2022Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answeringcited by 24
  16. 2024Can We Talk Models Into Seeing the World Differently?cited by 23
  17. 2024Factorized Diffusion: Perceptual Illusions by Noise Decompositioncited by 23
  18. 2024Compositional Generative Inverse Designcited by 20
  19. 2023Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignmentcited by 18
  20. 2024Can Models Learn Skill Composition from Examples?cited by 17
  21. 2023From Text to Mask: Localizing Entities Using the Attention of Text-to-Image Diffusion Modelscited by 16
  22. 2024Compositional Risk Minimizationcited by 13
  23. 2025Mechanisms of Projective Composition of Diffusion Modelscited by 13
  24. 2024Compositional Image Decomposition with Diffusion Modelscited by 12
  25. 2022vMFNet: Compositionality Meets Domain-generalised Segmentationcited by 11
  26. 2022Reasoning about Actions over Visual and Linguistic Modalities: A Surveycited by 10
  27. 2025Multi-Modal Manipulation via Multi-Modal Policy Consensuscited by 9
  28. 2025FreeBlend: Advancing Concept Blending with Staged Feedback-Driven Interpolation Diffusioncited by 8
  29. 2025CoInD: Enabling Logical Compositions in Diffusion Modelscited by 8
  30. 2022Markup-to-Image Diffusion Models with Scheduled Samplingcited by 6
  31. 2025SUB: Benchmarking CBM Generalization via Synthetic Attribute Substitutionscited by 6
  32. 2023Inferring Relational Potentials in Interacting Systemscited by 5
  33. 2024Instilling Multi-round Thinking to Text-guided Image Generationcited by 5
  34. 2024Text-to-Image Cross-Modal Generation: A Systematic Reviewcited by 5
  35. 2024Going beyond Compositions, DDPMs Can Produce Zero-Shot Interpolationscited by 5
  36. 2025Modality-Composable Diffusion Policy via Inference-Time Distribution-level Compositioncited by 5
  37. 2022DT2I: Dense Text-to-Image Generation from Region Descriptionscited by 4
  38. 2024Do Diffusion Models Learn Semantically Meaningful and Efficient Representations?cited by 4

50 most cited of 72 All 72 (JSON)