mlsanityData to 14 Oct 2026

Unsupervised Learning of Compositional Energy Concepts

arXiv 2111.03042 ↗First posted 4 Nov 2021cs.CV cs.AI cs.LGcited by 71

Not on the map. Why?

Cited by 23 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Humans are able to rapidly understand scenes by utilizing concepts extracted from prior experience. Such concepts are diverse, and include global scene descriptors, such as the weather or lighting, as well as local scene descriptors, such as the color or size of a particular object. So far, unsupervised discovery of concepts has focused on either modeling the global scene-level or the local object-level factors of variation, but not both. In this work, we propose COMET, which discovers and represents concepts as separate energy functions, enabling us to represent both global concepts as well as objects under a unified framework. COMET discovers energy functions through recomposing the input image, which we find captures independent factors without additional supervision. Sample generation in COMET is formulated as an optimization process on underlying energy functions, enabling us to generate images with permuted and composed concepts. Finally, discovered visual concepts in COMET generalize well, enabling us to compose concepts between separate modalities of images as well as with other concepts discovered by a separate instance of COMET trained on a different dataset. Code and data available at https://energy-based-model.github.io/comet/.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Builds on 38 of its 54 references that are in the map. First posted on arXiv 4 Nov 2021.

Builds on

38 of 54 references.

References · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  4. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  6. 2020Object-Centric Learning with Slot Attentioncited by 876
  7. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 503
  8. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
  9. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 392
  10. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 360
  11. 2020Virtual KITTI 2cited by 353
  12. 2018An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solutioncited by 307
26 more
  1. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 282
  2. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 273
  3. 2020Weakly-Supervised Disentanglement Without Compromisescited by 239
  4. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
  5. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  6. 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
  7. 2016A Theory of Generative ConvNetcited by 174
  8. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 130
  9. 2018Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objectscited by 128
  10. 2017Neural Expectation Maximizationcited by 123
  11. 2020Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Codingcited by 122
  12. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  13. 2019On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Modelscited by 105
  14. 2020On Linear Identifiability of Learned Representationscited by 105
  15. 2020ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICAcited by 102
  16. 2018Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactionscited by 100
  17. 2018Variational Autoencoders Pursue PCA Directions (by Accident)cited by 85
  18. 2019Flow Contrastive Estimation of Energy-Based Modelscited by 75
  19. 2019Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoningcited by 61
  20. 2017Generative Models of Visually Grounded Imaginationcited by 44
  21. 2020Semi-Supervised StyleGAN for Disentanglement Learningcited by 43
  22. 2016Deep Directed Generative Models with Energy-Based Probability Estimationcited by 34
  23. 2020Unsupervised Discovery of 3D Physical Objects from Videocited by 27
  24. 2019R-SQAIR: Relational Sequential Attend, Infer, Repeatcited by 21
  25. 2018Generative Modeling by Inclusive Neural Random Fields with Applications in Image Generation and Anomaly Detectioncited by 9
  26. 2018Investigating Object Compositionality in Generative Adversarial Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 14; 2023: 24; 2024: 18; 2025: 8; 2026: 5.

71 papers.

Sort:

By area:

  1. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  2. 2023Erasing Concepts from Diffusion Modelscited by 539
  3. 2021Conditional Object-Centric Learning from Videocited by 248
  4. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 199
  5. 2023One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applicationscited by 169
  6. 2022Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videoscited by 139
  7. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 93
  8. 2023Object-Centric Slot Diffusioncited by 90
  9. 2022Improving Object-centric Learning with Query Optimizationcited by 60
  10. 2023Concept Algebra for (Score-Based) Text-Controlled Generative Modelscited by 59
  11. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  12. 2023Self-supervised Object-Centric Learning for Videoscited by 49
38 more
  1. 2024Compositional Generative Modeling: A Single Model is Not All You Needcited by 41
  2. 2022Complex-Valued Autoencoders for Object Discoverycited by 40
  3. 2022Neural Systematic Bindercited by 37
  4. 2022PartAfford: Part-level Affordance Discovery from 3D Objectscited by 36
  5. 2023Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Modelscited by 36
  6. 2024Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretationscited by 33
  7. 2025Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptationcited by 33
  8. 2022Learning Iterative Reasoning through Energy Minimizationcited by 32
  9. 2023DisDiff: Unsupervised Disentanglement of Diffusion Probabilistic Modelscited by 30
  10. 2023Rotating Features for Object Discoverycited by 29
  11. 2023Probabilistic Adaptation of Text-to-Video Modelscited by 29
  12. 2023Compositional Sculpting of Iterative Generative Processescited by 27
  13. 2024Diffusion Model with Cross Attention as an Inductive Bias for Disentanglementcited by 27
  14. 2021Interactive Disentanglement: Learning Concepts by Interacting with their Prototype Representationscited by 24
  15. 2022Neural Groundplans: Persistent Neural Scene Representations from a Single Imagecited by 17
  16. 2024Learning Discrete Concepts in Latent Hierarchical Modelscited by 17
  17. 2022ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Timecited by 16
  18. 2025Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gatecited by 16
  19. 2022Visual Concepts Tokenizationcited by 14
  20. 2024Compositional Risk Minimizationcited by 13
  21. 2024Compositional Image Decomposition with Diffusion Modelscited by 12
  22. 2024SlotLifter: Slot-guided Feature Lifting for Learning Object-centric Radiance Fieldscited by 12
  23. 2023Hybrid Energy Based Model in the Feature Space for Out-of-Distribution Detectioncited by 9
  24. 2025Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Modelscited by 8
  25. 2022Robust and Controllable Object-Centric Learning through Energy-based Modelscited by 7
  26. 2023Energy-Inspired Self-Supervised Pretraining for Vision Modelscited by 7
  27. 2023Geometry of Score Based Generative Modelscited by 7
  28. 2023Vector-based Representation is the Key: A Study on Disentanglement and Compositional Generalizationcited by 6
  29. 2024Preference Optimization for Molecule Synthesis with Conditional Residual Energy-based Modelscited by 6
  30. 2024EQ-CBM: A Probabilistic Concept Bottleneck with Energy-based Models and Quantized Vectorscited by 6
  31. 2023Inferring Relational Potentials in Interacting Systemscited by 5
  32. 2024Binding Dynamics in Rotating Featurescited by 5
  33. 2024Bridging Associative Memory and Probabilistic Modelingcited by 5
  34. 2024Going beyond Compositions, DDPMs Can Produce Zero-Shot Interpolationscited by 5
  35. 2024Swing-by Dynamics in Concept Learning and Compositional Generalizationcited by 5
  36. 2023Im-Promptu: In-Context Composition from Image Promptscited by 4
  37. 2023Benchmarking and Analysis of Unsupervised Object Segmentation from Real-world Single Imagescited by 4
  38. 2024Do Diffusion Models Learn Semantically Meaningful and Efficient Representations?cited by 4

50 most cited of 71 All 71 (JSON)