What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectives
Not on the map. Why?
Abstract (arXiv)
Compositional generalization, the ability to generate novel combinations of known concepts, is a key ingredient for visual generative models. Yet, not all mechanisms that enable or inhibit it are fully understood. In this work, we conduct a systematic study of which design choices critically determine compositional generalization in image and video generation. By isolating independent design axes, we identify two key factors strongly associated with compositional success: (i) whether the training objective operates on a discrete or continuous distribution, and (ii) the completeness of conditioning information about constituent factors during training. We also show that relaxing the discrete loss with an auxiliary continuous latent objective can partially recover compositional performance in discrete models like MaskGIT. Our findings, corroborated by diverse compositional tasks and preliminary evidence in world models and LLMs, motivate a shift toward continuous objectives for compositional generalization.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 59 of its 77 references that are in the map. First posted on arXiv 3 Oct 2025.
Builds on
59 of 77 references.
References · 59
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2022Scalable Diffusion Models with Transformerscited by 5,931
- 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
- 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2022Video Diffusion Modelscited by 2,280
- 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
- 2021Variational Diffusion Modelscited by 1,145
- 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
38 more
- 2022MaskGIT: Masked Generative Image Transformercited by 1,082
- 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
- 2022Toy Models of Superpositioncited by 928
- 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
- 2024Emu3: Next-Token Prediction is All You Needcited by 761
- 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
- 2024Genie: Generative Interactive Environmentscited by 706
- 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
- 2024Autoregressive Image Generation without Vector Quantizationcited by 644
- 2021Knowledge Neurons in Pretrained Transformerscited by 590
- 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
- 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
- 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
- 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
- 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
- 2022Kubric: A scalable dataset generatorcited by 373
- 2018Towards a Definition of Disentangled Representationscited by 294
- 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
- 2013Direct and Indirect Effectscited by 276
- 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 218
- 2021Generative Adversarial Transformerscited by 120
- 2024An analytic theory of creativity in convolutional diffusion modelscited by 103
- 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
- 2023GIVT: Generative Infinite-Vocabulary Transformerscited by 80
- 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
- 2023Compositional Generalization from First Principlescited by 62
- 2023Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolationcited by 56
- 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
- 2023Provable Compositional Generalization for Object-Centric Learningcited by 41
- 2019Symmetry-Based Disentangled Representation Learning requires Interaction with Environmentscited by 35
- 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
- 2025How Compositional Generalization and Creativity Improve as Diffusion Models are Trainedcited by 24
- 2022Toward Compositional Generalization in Object-Oriented World Modelingcited by 22
- 2024Beyond FVD: Enhanced Evaluation Metrics for Video Generation Qualitycited by 21
- 2024Interaction Asymmetry: A General Principle for Learning Composable Abstractionscited by 19
- 2025Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Modelscited by 17
- 2023MAGVLT: Masked Generative Vision-and-Language Transformercited by 14
50 most cited of 59 All 59 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.