# Interaction Asymmetry: A General Principle for Learning Composable Abstractions

arXiv: https://arxiv.org/abs/2411.07784 · HTML: https://mlsanity.com/paper/2411.07784/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 124 references that are in the map.
- First posted on arXiv 12 Nov 2024.

## Builds on (most cited first, 30 of 65)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [Scalable Adaptive Computation for Iterative Generation](https://mlsanity.com/paper/2212.11972/) (2022)
- [Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA](https://mlsanity.com/paper/2107.10098/) (2021)
- [Illiterate DALL-E Learns to Compose](https://mlsanity.com/paper/2110.11405/) (2021)
- [Interventional Causal Representation Learning](https://mlsanity.com/paper/2209.11924/) (2022)
- [Weakly supervised causal representation learning](https://mlsanity.com/paper/2203.16437/) (2022)
- [Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos](https://mlsanity.com/paper/2205.14065/) (2022)
- [CITRIS: Causal Identifiability from Temporal Intervened Sequences](https://mlsanity.com/paper/2202.03169/) (2022)
- [Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding](https://mlsanity.com/paper/2007.10930/) (2020)
- [Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs](https://mlsanity.com/paper/1901.07017/) (2019)
- [Nonparametric Identifiability of Causal Representations from Unknown Interventions](https://mlsanity.com/paper/2306.00542/) (2023)
- [Independent mechanism analysis, a new concept?](https://mlsanity.com/paper/2106.05200/) (2021)

## Built on by (most cited first, 20 of 20)

- [Are We Done with Object-Centric Learning?](https://mlsanity.com/paper/2504.07092/) (2025)
- [Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research](https://mlsanity.com/paper/2504.13101/) (2025)
- [Scaling can lead to compositional generalization](https://mlsanity.com/paper/2507.07207/) (2025)
- [From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?](https://mlsanity.com/paper/2512.15134/) (2025)
- [Diverse Influence Component Analysis: A Geometric Approach to Nonlinear Mixture Identifiability](https://mlsanity.com/paper/2510.17040/) (2025)
- [Mechanistic Independence: A Principle for Identifiable Disentangled Representations](https://mlsanity.com/paper/2509.22196/) (2025)
- [What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectives](https://mlsanity.com/paper/2510.03075/) (2025)
- [Content-Style Identification via Differential Independence](https://mlsanity.com/paper/2605.17827/) (2026)
- [A Compressive-Expressive Communication Framework for Compositional Representations](https://mlsanity.com/paper/2501.19182/) (2025)
- [Is Generation Required for Data-Efficient Perception?](https://mlsanity.com/paper/2512.08854/) (2025)
- [Learning by Analogy: A Causal Framework for Composition Generalization](https://mlsanity.com/paper/2512.10669/) (2025)
- [Extrapolation Guarantees for Perturbation Modeling Under the Additive Latent Shift Assumption](https://mlsanity.com/paper/2504.18522/) (2025)
- [A Theoretical Analysis of Provable Compositional Generalization in Neural Networks: A Necessary and Sufficient Condition](https://mlsanity.com/paper/2505.02627/) (2025)
- [Eigenfunction Extraction for Ordered Representation Learning](https://mlsanity.com/paper/2510.24672/) (2025)
- [Disentanglement Beyond Generative Models with Riemannian ICA](https://mlsanity.com/paper/2605.22531/) (2026)
- [A mathematical theory of balancing relational generalization and memorization](https://mlsanity.com/paper/2605.22972/) (2026)
- [CenSynCMB: Centre Maps and Physics-Guided Synthesis for Microbleed Detection](https://mlsanity.com/paper/2607.05325/) (2026)
- [Beyond ICA: Identifiability by Symmetry Breaking](https://mlsanity.com/paper/2607.23182/) (2026)
- [Towards Identifiable Representations under Misspecified Structure](https://mlsanity.com/paper/2609.33273/) (2026)
- [Predicting Unseen Scientific Interactions: Conditional Flow Models Hallucinate Non-Additive Effects](https://mlsanity.com/paper/2610.20288/) (2026)

## Abstract (arXiv)

Learning disentangled representations of concepts and re-composing them in unseen ways is crucial for generalizing to out-of-domain situations. However, the underlying properties of concepts that enable such disentanglement and compositional generalization remain poorly understood. In this work, we propose the principle of interaction asymmetry which states: "Parts of the same concept have more complex interactions than parts of different concepts". We formalize this via block diagonality conditions on the (n+1)th order derivatives of the generator mapping concepts to observed data, where different orders of "complexity" correspond to different n. Using this formalism, we prove that interaction asymmetry enables both disentanglement and compositional generalization. Our results unify recent theoretical results for learning concepts of objects, which we show are recovered as special cases with n!=!0 or 1. We provide results for up to n!=!2, thus extending these prior works to more flexible generator functions, and conjecture that the same proof strategies generalize to larger n. Practically, our theory suggests that, to disentangle concepts, an autoencoder should penalize its latent capacity and the interactions between concepts during decoding. We propose an implementation of these criteria using a flexible Transformer-based VAE, with a novel regularizer on the attention weights of the decoder. On synthetic image datasets consisting of objects, we provide evidence that this model can achieve comparable object disentanglement to existing models that use more explicit object-centric priors.