mlsanityData to 7 Oct 2026

Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation

arXiv 2307.02598 ↗First posted 5 Jul 2023cs.LG stat.MLcited by 56

Not on the map. Why?

Abstract (arXiv)

We tackle the problems of latent variables identification and ``out-of-support'' image generation in representation learning. We show that both are possible for a class of decoders that we call additive, which are reminiscent of decoders used for object-centric representation learning (OCRL) and well suited for images that can be decomposed as a sum of object-specific images. We provide conditions under which exactly solving the reconstruction problem using an additive decoder is guaranteed to identify the blocks of latent variables up to permutation and block-wise invertible transformations. This guarantee relies only on very weak assumptions about the distribution of the latent factors, which might present statistical dependencies and have an almost arbitrarily shaped support. Our result provides a new setting where nonlinear independent component analysis (ICA) is possible and adds to our theoretical understanding of OCRL methods. We also show theoretically that additive decoders can generate novel images by recombining observed factors of variations in novel ways, an ability we refer to as Cartesian-product extrapolation. We show empirically that additivity is crucial for both identifiability and extrapolation on simulated data.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 48 of its 71 references that are in the map. First posted on arXiv 5 Jul 2023.

Builds on

48 of 71 references.

References · 48

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  4. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  5. 2020Object-Centric Learning with Slot Attentioncited by 865
  6. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  7. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
  8. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  9. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
  10. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  11. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 278
  12. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 273
36 more
  1. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 270
  2. 2020Neurosymbolic AI: The 3rd Wavecited by 253
  3. 2020Weakly-Supervised Disentanglement Without Compromisescited by 237
  4. 2020On the Binding Problem in Artificial Neural Networkscited by 230
  5. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
  6. 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
  7. 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
  8. 2021Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICAcited by 158
  9. 2022Weakly supervised causal representation learningcited by 140
  10. 2022Interventional Causal Representation Learningcited by 140
  11. 2022CITRIS: Causal Identifiability from Temporal Intervened Sequencescited by 132
  12. 2017Neural Expectation Maximizationcited by 123
  13. 2020Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Codingcited by 121
  14. 2023Nonparametric Identifiability of Causal Representations from Unknown Interventionscited by 105
  15. 2020ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICAcited by 102
  16. 2021Independent mechanism analysis, a new concept?cited by 96
  17. 2023Learning Linear Causal Representations from Interventions under General Nonlinear Mixingcited by 92
  18. 2020The Hessian Penalty: A Weak Prior for Unsupervised Disentanglementcited by 90
  19. 2022On the Identifiability of Nonlinear ICA: Sparsity and Beyondcited by 84
  20. 2022Linear Causal Disentanglement via Interventionscited by 84
  21. 2021Generalization and Robustness Implications in Object-Centric Learningcited by 81
  22. 2021Desiderata for Representation Learning: A Causal Perspectivecited by 76
  23. 2016Tagger: Deep Unsupervised Perceptual Groupingcited by 69
  24. 2021Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICAcited by 64
  25. 2022Weakly Supervised Representation Learning with Sparse Perturbationscited by 64
  26. 2021Identifiable Deep Generative Models via Sparse Decodingcited by 54
  27. 2022Function Classes for Identifiable Nonlinear Independent Component Analysiscited by 53
  28. 2023Provably Learning Object-Centric Representationscited by 45
  29. 2022Disentanglement of Correlated Factors via Hausdorff Factorized Supportcited by 37
  30. 2020Learning Representations that Support Extrapolationcited by 36
  31. 2023Learning nonparametric latent causal graphs with unknown interventionscited by 34
  32. 2022Indeterminacy in Generative Models: Characterization and Strong Identifiabilitycited by 33
  33. 2021Properties from Mechanisms: An Equivariance Perspective on Identifiable Representation Learningcited by 30
  34. 2022Partial Disentanglement via Mechanism Sparsitycited by 29
  35. 2022Embrace the Gap: VAEs Perform Independent Mechanism Analysiscited by 28
  36. 2020A theory of independent mechanisms for extrapolation in generative modelscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 18; 2025: 21; 2026: 14.

56 papers.

Sort:

By area:

  1. 2023Provable Compositional Generalization for Object-Centric Learningcited by 41
  2. 2024Compositional Generative Modeling: A Single Model is Not All You Needcited by 41
  3. 2023Multi-Domain Causal Representation Learning via Weak Distributional Invariancescited by 23
  4. 2024On Provable Length and Compositional Generalizationcited by 22
  5. 2024Marrying Causal Representation Learning with Dynamical Systems for Sciencecited by 22
  6. 2023Object-centric architectures enable efficient causal representation learningcited by 21
  7. 2024Position: Understanding LLMs Requires More Than Statistical Generalizationcited by 20
  8. 2024When does compositional structure yield compositional generalization? A kernel theorycited by 19
  9. 2024Interaction Asymmetry: A General Principle for Learning Composable Abstractionscited by 19
  10. 2024Learning Discrete Concepts in Latent Hierarchical Modelscited by 17
  11. 2024Identifiable Exchangeable Mechanisms for Causal Structure and Representation Learningcited by 16
  12. 2024On the Identification of Temporally Causal Representation with Instantaneous Dependencecited by 15
38 more
  1. 2025SmartCLIP: Modular Vision-language Alignment with Identification Guaranteescited by 14
  2. 2024Compositional Risk Minimizationcited by 13
  3. 2024A Complexity-Based Theory of Compositionalitycited by 13
  4. 2024Identifiable Object-Centric Representation Learning via Probabilistic Slot Attentioncited by 9
  5. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  6. 2024Language Agents Meet Causality -- Bridging LLMs and Causal World Modelscited by 8
  7. 2026Causality is Key for Interpretability Claims to Generalisecited by 8
  8. 2024Binding Dynamics in Rotating Featurescited by 5
  9. 2025From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?cited by 5
  10. 2026Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisationcited by 5
  11. 2024Toward the Identifiability of Comparative Deep Generative Modelscited by 4
  12. 2025Towards Understanding Extrapolation: a Causal Lenscited by 4
  13. 2025unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoningcited by 4
  14. 2025Diverse Influence Component Analysis: A Geometric Approach to Nonlinear Mixture Identifiabilitycited by 4
  15. 2025Mechanistic Independence: A Principle for Identifiable Disentangled Representationscited by 3
  16. 2025Nonparametric Identification of Latent Conceptscited by 3
  17. 2025What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectivescited by 3
  18. 2026Unsupervised Decomposition and Recombination with Discriminator-Driven Diffusion Modelscited by 3
  19. 2026Disentangling Dynamical Systems: Causal Representation Learning Meets Local Sparse Attentioncited by 3
  20. 2024Rule Extrapolation in Language Models: A Study of Compositional Generalization on OOD Promptscited by 2
  21. 2025Supervised Contrastive Block Disentanglementcited by 2
  22. 2024Linear causal disentanglement via higher-order cumulantscited by 1
  23. 2025A Compressive-Expressive Communication Framework for Compositional Representationscited by 1
  24. 2025Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priorscited by 1
  25. 2025Disentangled Representation Learning via Modular Compositional Biascited by 1
  26. 2025Is Generation Required for Data-Efficient Perception?cited by 1
  27. 2025Learning by Analogy: A Causal Framework for Composition Generalizationcited by 1
  28. 2026Identifiability of Potentially Degenerate Gaussian Mixture Models With Piecewise Affine Mixingcited by 1
  29. 2026Content-Style Identification via Differential Independencecited by 1
  30. 2024Disentanglement as Identifiable Pushforward Factorisationcited by 0
  31. 2025Time-varying Factor Augmented Vector Autoregression with Grouped Sparse Autoencodercited by 0
  32. 2025Extrapolation Guarantees for Perturbation Modeling Under the Additive Latent Shift Assumptioncited by 0
  33. 2025Learning Joint Interventional Effects from Single-Variable Interventions in Additive Modelscited by 0
  34. 2025Identifiable Object Representations under Spatial Ambiguitiescited by 0
  35. 2025Operationalizing Quantized Disentanglementcited by 0
  36. 2025Beyond the Black Box: Identifiable Interpretation and Control in Generative Models via Causal Minimalitycited by 0
  37. 2026Meta-probabilistic Modelingcited by 0
  38. 2026SEDGE: Structural Extrapolated Data Generationcited by 0

50 most cited of 56 All 56 (JSON)