# Compositional Visual Generation and Inference with Energy Based Models

arXiv: https://arxiv.org/abs/2004.06030 · HTML: https://mlsanity.com/paper/2004.06030/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2020.

## Built on by (most cited first, 26 of 26)

- [Learning to Compose Visual Relations](https://mlsanity.com/paper/2111.09297/) (2021)
- [ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty](https://mlsanity.com/paper/2408.14339/) (2024)
- [Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations](https://mlsanity.com/paper/2408.04380/) (2024)
- [All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2312.12807/) (2023)
- [VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming](https://mlsanity.com/paper/2202.04178/) (2022)
- [Composable Text Controls in Latent Space with ODEs](https://mlsanity.com/paper/2208.00638/) (2022)
- [Can Models Learn Skill Composition from Examples?](https://mlsanity.com/paper/2409.19808/) (2024)
- [Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning](https://mlsanity.com/paper/2105.04962/) (2021)
- [ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Time](https://mlsanity.com/paper/2206.15049/) (2022)
- [Mechanisms of Projective Composition of Diffusion Models](https://mlsanity.com/paper/2502.04549/) (2025)
- [Exploring Compositional Visual Generation with Latent Classifier Guidance](https://mlsanity.com/paper/2304.12536/) (2023)
- [Going beyond Compositions, DDPMs Can Produce Zero-Shot Interpolations](https://mlsanity.com/paper/2405.19201/) (2024)
- [SecureLLM: Using Compositionality to Build Provably Secure Language Models for Private, Sensitive, and Secret Data](https://mlsanity.com/paper/2405.09805/) (2024)
- [Few-Shot Task Learning through Inverse Generative Modeling](https://mlsanity.com/paper/2411.04987/) (2024)
- [Tree-Guided Diffusion Planner](https://mlsanity.com/paper/2508.21800/) (2025)
- [Versatile Energy-Based Probabilistic Models for High Energy Physics](https://mlsanity.com/paper/2302.00695/) (2023)
- [Steering Language Generation: Harnessing Contrastive Expert Guidance and Negative Prompting for Coherent and Diverse Synthetic Data Generation](https://mlsanity.com/paper/2308.07645/) (2023)
- [Generating by Understanding: Neural Visual Generation with Logical Symbol Groundings](https://mlsanity.com/paper/2310.17451/) (2023)
- [When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Models](https://mlsanity.com/paper/2601.11444/) (2026)
- [Test-Time Compositional Generalization in Diffusion Models via Concept Discovery](https://mlsanity.com/paper/2605.07078/) (2026)
- [Asynchronous Multimodal Diffusion Policy Composition via Latency-Aware Guidance Fusion](https://mlsanity.com/paper/2607.17257/) (2026)
- [Energy-based models for diagnostic reconstruction and analysis in a laboratory plasma device](https://mlsanity.com/paper/2605.08645/) (2026)
- [Compositional Video Generation via Inference-Time Guidance](https://mlsanity.com/paper/2605.14988/) (2026)
- [StyleForge: Indoor Furniture Styling by Counterfactual Reasoning in a Hypergraph Field](https://mlsanity.com/paper/2608.01954/) (2026)
- [Latent Recurrent Thoughts: Recurrent Refinement of Proposed Latents for Reasoning with Frozen LLMs](https://mlsanity.com/paper/2609.01117/) (2026)
- [Spacecraft Rendezvous Trajectory Generation with Modular Constraints via Diffusion Model Composition](https://mlsanity.com/paper/2610.05642/) (2026)

## Abstract (arXiv)

A vital aspect of human intelligence is the ability to compose increasingly complex concepts out of simpler ideas, enabling both rapid learning and adaptation of knowledge. In this paper we show that energy-based models can exhibit this ability by directly combining probability distributions. Samples from the combined distribution correspond to compositions of concepts. For example, given a distribution for smiling faces, and another for male faces, we can combine them to generate smiling male faces. This allows us to generate natural images that simultaneously satisfy conjunctions, disjunctions, and negations of concepts. We evaluate compositional generation abilities of our model on the CelebA dataset of natural faces and synthetic 3D scene images. We also demonstrate other unique advantages of our model, such as the ability to continually learn and incorporate new concepts, or infer compositions of concept properties underlying an image.