mlsanityData to 7 Oct 2026

On Linear Identifiability of Learned Representations

arXiv 2007.00810 ↗First posted 1 Jul 2020stat.ML cs.LGcited by 103

Not on the map. Why?

Abstract (arXiv)

Identifiability is a desirable property of a statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational resources and data. We study identifiability in the context of representation learning: discovering nonlinear data representations that are optimal with respect to some downstream task. When parameterized as deep neural networks, such representation functions typically lack identifiability in parameter space, because they are overparameterized by design. In this paper, building on recent advances in nonlinear ICA, we aim to rehabilitate identifiability by showing that a large family of discriminative models are in fact identifiable in function space, up to a linear indeterminacy. Many models for representation learning in a wide variety of domains have been identifiable in this sense, including text, images and audio, state-of-the-art at time of publication. We derive sufficient conditions for linear identifiability and provide empirical support for the result on both simulated and real-world data.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 7; 2023: 17; 2024: 18; 2025: 19; 2026: 30.

103 papers.

Sort:

By area:

  1. 2024The Platonic Representation Hypothesiscited by 362
  2. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  3. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  4. 2021Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICAcited by 158
  5. 2022Learnable latent embeddings for joint behavioral and neural analysiscited by 113
  6. 2021Generalized Shape Metrics on Neural Representationscited by 109
  7. 2023Nonparametric Identifiability of Causal Representations from Unknown Interventionscited by 105
  8. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  9. 2023Learning Linear Causal Representations from Interventions under General Nonlinear Mixingcited by 92
  10. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  11. 2021Unsupervised Learning of Compositional Energy Conceptscited by 71
  12. 2022Identifiability of deep generative models without auxiliary informationcited by 70
38 more
  1. 2021Visual Representation Learning Does Not Generalize Strongly Within the Same Domaincited by 60
  2. 2022Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learningcited by 44
  3. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 39
  4. 2023Learning nonparametric latent causal graphs with unknown interventionscited by 34
  5. 2023Unpaired Multi-Domain Causal Representation Learningcited by 32
  6. 2024T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitchingcited by 32
  7. 2023Causal Representation Learning Made Identifiable by Grouping of Observational Variablescited by 30
  8. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 30
  9. 2026When Does LeJEPA Learn a World Model?cited by 27
  10. 2024On Provable Length and Compositional Generalizationcited by 22
  11. 2021I Don't Need u: Identifiable Non-Linear ICA Without Side Informationcited by 21
  12. 2023Identifying Representations for Intervention Extrapolationcited by 21
  13. 2023The Emergence of Reproducibility and Generalizability in Diffusion Modelscited by 21
  14. 2023Identifying Linearly-Mixed Causal Representations from Multi-Node Interventionscited by 18
  15. 2022Towards efficient representation identification in supervised learningcited by 14
  16. 2023Identifiable Contrastive Learning with Automatic Feature Importance Discoverycited by 14
  17. 2023Learning Causal Representations from General Environments: Identifiability and Intrinsic Ambiguitycited by 14
  18. 2024All or None: Identifiable Linear Properties of Next-token Predictors in Language Modelingcited by 14
  19. 2026InfoNCE Induces Gaussian Distributioncited by 14
  20. 2025Shortcuts and Identifiability in Concept-based Models from a Neuro-Symbolic Lenscited by 13
  21. 2025I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?cited by 13
  22. 2024Identifiable Causal Representation Learning: Unsupervised, Multi-View, and Multi-Environmentcited by 12
  23. 2025Identifiability of Deep Polynomial Neural Networkscited by 12
  24. 2024Evaluating Representational Similarity Measures from the Lens of Functional Correspondencecited by 11
  25. 2021Intact-VAE: Estimating Treatment Effects under Unobserved Confoundingcited by 10
  26. 2024Formation of Representations in Neural Networkscited by 10
  27. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 10
  28. 2024Self-supervised contrastive learning performs non-linear system identificationcited by 9
  29. 2024Automated Discovery of Pairwise Interactions from Unstructured Datacited by 8
  30. 2025Diffusion Counterfactual Generation with Semantic Abductioncited by 8
  31. 2026Causality is Key for Interpretability Claims to Generalisecited by 8
  32. 2023Measuring the Instability of Fine-Tuningcited by 7
  33. 2025Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent Spacescited by 6
  34. 2025Quantifying Structure in CLIP Embeddings: A Statistical Framework for Concept Interpretationcited by 6
  35. 2023Towards understanding neural collapse in supervised contrastive learning with the information bottleneck methodcited by 5
  36. 2023Invariance & Causal Representation Learning: Prospects and Limitationscited by 5
  37. 2026Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisationcited by 5
  38. 2021Towards Principled Causal Effect Estimation by Deep Identifiable Modelscited by 4

50 most cited of 103 All 103 (JSON)