mlsanityData to 14 Oct 2026

Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning

arXiv 2211.14666 ↗First posted 26 Nov 2022cs.LG stat.MLcited by 44

Not on the map. Why?

Abstract (arXiv)

Although disentangled representations are often said to be beneficial for downstream tasks, current empirical and theoretical understanding is limited. In this work, we provide evidence that disentangled representations coupled with sparse base-predictors improve generalization. In the context of multi-task learning, we prove a new identifiability result that provides conditions under which maximally sparse base-predictors yield disentangled representations. Motivated by this theoretical result, we propose a practical approach to learn disentangled representations based on a sparsity-promoting bi-level optimization problem. Finally, we explore a meta-learning version of this algorithm based on group Lasso multiclass SVM base-predictors, for which we derive a tractable dual formulation. It obtains competitive results on standard few-shot classification benchmarks, while each task is using only a fraction of the learned representations.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 61 of its 95 references that are in the map. First posted on arXiv 26 Nov 2022.

Builds on

61 of 95 references.

References · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2016Layer Normalizationcited by 5,669
  9. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  10. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  11. 2021Convex optimizationcited by 2,515
  12. 2016Matching Networks for One Shot Learningcited by 2,255
38 more
  1. 2019Invariant Risk Minimizationcited by 1,854
  2. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  3. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  4. 2021Robust fine-tuning of zero-shot modelscited by 797
  5. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 742
  6. 2018Disentangling by Factorisingcited by 616
  7. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 503
  8. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  9. 2018Meta-learning with differentiable closed-form solverscited by 397
  10. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  11. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 282
  12. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 275
  13. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 273
  14. 2019Causality for Machine Learningcited by 243
  15. 2017Variational Inference of Disentangled Latent Concepts from Unlabeled Observationscited by 241
  16. 2020Weakly-Supervised Disentanglement Without Compromisescited by 239
  17. 2016Hyperparameter optimization with approximate gradientcited by 233
  18. 2021Towards Causal Representation Learningcited by 209
  19. 2021Efficient and Modular Implicit Differentiationcited by 203
  20. 2015The Benefit of Multitask Representation Learningcited by 195
  21. 2020Invariant Risk Minimization Gamescited by 188
  22. 2018Isolating Sources of Disentanglement in Variational Autoencoderscited by 167
  23. 2021Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICAcited by 160
  24. 2022Weakly supervised causal representation learningcited by 140
  25. 2022CITRIS: Causal Identifiability from Temporal Intervened Sequencescited by 133
  26. 2020Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Codingcited by 122
  27. 2020On Linear Identifiability of Learned Representationscited by 105
  28. 2020ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICAcited by 102
  29. 2017Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observationscited by 100
  30. 2019Are Disentangled Representations Helpful for Abstract Visual Reasoning?cited by 100
  31. 2021Learning Temporally Causal Latent Processes from General Temporal Datacited by 98
  32. 2021Independent mechanism analysis, a new concept?cited by 97
  33. 2021Towards Principled Disentanglement for Domain Generalizationcited by 91
  34. 2022On the Identifiability of Nonlinear ICA: Sparsity and Beyondcited by 85
  35. 2022A very preliminary analysis of DALL-E 2cited by 82
  36. 2022Weakly Supervised Representation Learning with Sparse Perturbationscited by 65
  37. 2020On the Transfer of Disentangled Representations in Realistic Settingscited by 63
  38. 2019Disentangling Factors of Variation Using Few Labelscited by 57

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 17; 2025: 12; 2026: 8.

44 papers.

Sort:

By area:

  1. 2023Nonparametric Identifiability of Causal Representations from Unknown Interventionscited by 106
  2. 2023Multi-View Causal Representation Learning with Partial Observabilitycited by 74
  3. 2023Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcutscited by 63
  4. 2024A Sparsity Principle for Partially Observable Causal Representation Learningcited by 35
  5. 2024Unifying Causal Representation Learning with the Invariance Principlecited by 31
  6. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 31
  7. 2024Marrying Causal Representation Learning with Dynamical Systems for Sciencecited by 23
  8. 2024Interaction Asymmetry: A General Principle for Learning Composable Abstractionscited by 20
  9. 2024A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcutscited by 19
  10. 2023Identifying Linearly-Mixed Causal Representations from Multi-Node Interventionscited by 18
  11. 2024The Landscape of Causal Discovery Data: Grounding Causal Discovery in Real-World Applicationscited by 18
  12. 2024Analyzing (In)Abilities of SAEs via Formal Languagescited by 17
32 more
  1. 2024On the Identification of Temporally Causal Representation with Instantaneous Dependencecited by 16
  2. 2025From superposition to sparse codes: interpretable representations in neural networkscited by 16
  3. 2024All or None: Identifiable Linear Properties of Next-token Predictors in Language Modelingcited by 14
  4. 2024A Complexity-Based Theory of Compositionalitycited by 13
  5. 2024Causal Representation Learning from Multimodal Biomedical Observationscited by 13
  6. 2025Shortcuts and Identifiability in Concept-based Models from a Neuro-Symbolic Lenscited by 13
  7. 2024Identifiable Causal Representation Learning: Unsupervised, Multi-View, and Multi-Environmentcited by 12
  8. 2025From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?cited by 7
  9. 2025What is causal about causal models and representations?cited by 6
  10. 2026Statistical and structural identifiability in representation learningcited by 6
  11. 2023Invariance & Causal Representation Learning: Prospects and Limitationscited by 5
  12. 2024Causal Representation Learning in Temporal Data via Single-Parent Decodingcited by 5
  13. 2025Sanity Checking Causal Representation Learning on a Simple Real-World Systemcited by 5
  14. 2024From Orthogonality to Dependency: Learning Disentangled Representation for Multi-Modal Time-Series Sensing Signalscited by 4
  15. 2024Disentangling Representations through Multi-task Learningcited by 4
  16. 2025The Third Pillar of Causal Analysis? A Measurement Perspective on Causal Representationscited by 4
  17. 2025When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspectivecited by 4
  18. 2026Disentangling Dynamical Systems: Causal Representation Learning Meets Local Sparse Attentioncited by 3
  19. 2024SGW-based Multi-Task Learning in Vision Taskscited by 2
  20. 2025A Dual-Task Synergy-Driven Generalization Framework for Pancreatic Cancer Segmentation in CT Scanscited by 2
  21. 2025Superclass-Guided Representation Disentanglement for Spurious Correlation Mitigationcited by 2
  22. 2025Online Time Series Forecasting with Theoretical Guaranteescited by 2
  23. 2023Leveraging Task Structures for Improved Identifiability in Neural Network Representationscited by 1
  24. 2025Causal View of Time Series Imputation: Some Identification Results on Missing Mechanismcited by 1
  25. 2026Hierarchical Action Learning for Weakly-Supervised Action Segmentationcited by 1
  26. 2026Identifiability of Potentially Degenerate Gaussian Mixture Models With Piecewise Affine Mixingcited by 1
  27. 2023SPARLING: Learning Latent Representations with Extremely Sparse Activationscited by 0
  28. 2025Beyond the Black Box: Identifiable Interpretation and Control in Generative Models via Causal Minimalitycited by 0
  29. 2026Logit Distance Bounds Representational Similaritycited by 0
  30. 2026CORE-MTL: Rethinking Gradient Balancing via Causal Orthogonal Representationscited by 0
  31. 2026Towards Identifiable Representations under Misspecified Structurecited by 0
  32. 2026Identifiability Guarantees for Drivers and Dynamics of Delayed Physical Systemscited by 0