# Towards efficient representation identification in supervised learning

arXiv: https://arxiv.org/abs/2204.04606 · HTML: https://mlsanity.com/paper/2204.04606/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 10 Apr 2022.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning](https://mlsanity.com/paper/1805.08651/) (2018)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Differentiable Causal Discovery from Interventional Data](https://mlsanity.com/paper/2007.01754/) (2020)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [Learning Neural Causal Models from Unknown Interventions](https://mlsanity.com/paper/1910.01075/) (2019)
- [On Linear Identifiability of Learned Representations](https://mlsanity.com/paper/2007.00810/) (2020)
- [ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA](https://mlsanity.com/paper/2002.11537/) (2020)
- [Weakly Supervised Disentanglement with Guarantees](https://mlsanity.com/paper/1910.09772/) (2019)
- [Desiderata for Representation Learning: A Causal Perspective](https://mlsanity.com/paper/2109.03795/) (2021)
- [Invariant Language Modeling](https://mlsanity.com/paper/2110.08413/) (2021)

## Built on by (most cited first, 14 of 14)

- [Interventional Causal Representation Learning](https://mlsanity.com/paper/2209.11924/) (2022)
- [Nonparametric Identifiability of Causal Representations from Unknown Interventions](https://mlsanity.com/paper/2306.00542/) (2023)
- [Score-based Causal Representation Learning with Interventions](https://mlsanity.com/paper/2301.08230/) (2023)
- [Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning](https://mlsanity.com/paper/2211.14666/) (2022)
- [Cross-Entropy Is All You Need To Invert the Data Generating Process](https://mlsanity.com/paper/2410.21869/) (2024)
- [Identifying Representations for Intervention Extrapolation](https://mlsanity.com/paper/2310.04295/) (2023)
- [Neuro-Symbolic Reasoning Shortcuts: Mitigation Strategies and their Limitations](https://mlsanity.com/paper/2303.12578/) (2023)
- [Learning Latent Structural Causal Models](https://mlsanity.com/paper/2210.13583/) (2022)
- [On the Identifiability of Quantized Factors](https://mlsanity.com/paper/2306.16334/) (2023)
- [Learning Invariant Causal Mechanism from Vision-Language Models](https://mlsanity.com/paper/2405.15289/) (2024)
- [Sparsity regularization via tree-structured environments for disentangled representations](https://mlsanity.com/paper/2405.20482/) (2024)
- [Inducing Causal Structure for Abstractive Text Summarization](https://mlsanity.com/paper/2308.12888/) (2023)
- [Learning Treatment Representations for Downstream Instrumental Variable Regression](https://mlsanity.com/paper/2506.02200/) (2025)
- [Incorporating Interventional Independence Improves Robustness against Interventional Distribution Shift](https://mlsanity.com/paper/2507.05412/) (2025)

## Abstract (arXiv)

Humans have a remarkable ability to disentangle complex sensory inputs (e.g., image, text) into simple factors of variation (e.g., shape, color) without much supervision. This ability has inspired many works that attempt to solve the following question: how do we invert the data generation process to extract those factors with minimal or no supervision? Several works in the literature on non-linear independent component analysis have established this negative result; without some knowledge of the data generation process or appropriate inductive biases, it is impossible to perform this inversion. In recent years, a lot of progress has been made on disentanglement under structural assumptions, e.g., when we have access to auxiliary information that makes the factors of variation conditionally independent. However, existing work requires a lot of auxiliary information, e.g., in supervised classification, it prescribes that the number of label classes should be at least equal to the total dimension of all factors of variation. In this work, we depart from these assumptions and ask: a) How can we get disentanglement when the auxiliary information does not provide conditional independence over the factors of variation? b) Can we reduce the amount of auxiliary information required for disentanglement? For a class of models where auxiliary information does not ensure conditional independence, we show theoretically and experimentally that disentanglement (to a large extent) is possible even when the auxiliary information dimension is much less than the dimension of the true latent representation.