mlsanityData to 7 Oct 2026

Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style

arXiv 2106.04619 ↗First posted 8 Jun 2021stat.ML cs.AI cs.CVcited by 308

Not on the map. Why?

Abstract (arXiv)

Self-supervised representation learning has shown remarkable success in a number of domains. A common practice is to perform data augmentation via hand-crafted transformations intended to leave the semantics of the data invariant. We seek to understand the empirical success of this approach from a theoretical perspective. We formulate the augmentation process as a latent variable model by postulating a partition of the latent representation into a content component, which is assumed invariant to augmentation, and a style component, which is allowed to change. Unlike prior work on disentanglement and independent component analysis, we allow for both nontrivial statistical and causal dependencies in the latent space. We study the identifiability of the latent representation based on pairs of views of the observations and prove sufficient conditions that allow us to identify the invariant content partition up to an invertible mapping in both generative and discriminative settings. We find numerical simulations with dependent latent variables are consistent with our theory. Lastly, we introduce Causal3DIdent, a dataset of high-dimensional, visually complex images with rich causal dependencies, which we use to study the effect of data augmentations performed in practice.

Key facts. Cited by 308 papers in the map (data through 7 Oct 2026). Builds on 76 of its 130 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

76 of 130 references.

References · 76

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2014Going Deeper with Convolutionscited by 4,231
  11. 2013Auto-Encoding Variational Bayescited by 3,980
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
38 more
  1. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  4. 2016Density estimation using Real NVPcited by 2,029
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  9. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  12. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  13. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  14. 2019Contrastive Multiview Codingcited by 1,297
  15. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  16. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  17. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  18. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  19. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  20. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  21. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  22. 2018Disentangling by Factorisingcited by 614
  23. 2019On Variational Bounds of Mutual Informationcited by 550
  24. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  25. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
  26. 2018Invertible Residual Networkscited by 364
  27. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  28. 2018Understanding disentangling in β-VAEcited by 332
  29. 2019On Mutual Information Maximization for Representation Learningcited by 315
  30. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 278
  31. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 270
  32. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
  33. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  34. 2019Causality for Machine Learningcited by 242
  35. 2017Variational Inference of Disentangled Latent Concepts from Unlabeled Observationscited by 240
  36. 2020Weakly-Supervised Disentanglement Without Compromisescited by 237
  37. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  38. 2021Towards Causal Representation Learningcited by 207

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 58; 2023: 81; 2024: 57; 2025: 56; 2026: 41.

308 papers.

Sort:

By area:

  1. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  2. 2021Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICAcited by 158
  3. 2022Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphscited by 144
  4. 2022Weakly supervised causal representation learningcited by 140
  5. 2022Interventional Causal Representation Learningcited by 140
  6. 2022CITRIS: Causal Identifiability from Temporal Intervened Sequencescited by 132
  7. 2023Nonparametric Identifiability of Causal Representations from Unknown Interventionscited by 105
  8. 2021Independent mechanism analysis, a new concept?cited by 96
  9. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  10. 2023Learning Linear Causal Representations from Interventions under General Nonlinear Mixingcited by 92
  11. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  12. 2023Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglementcited by 89
38 more
  1. 2023Multi-View Causal Representation Learning with Partial Observabilitycited by 72
  2. 2022Real-Time Style Modelling of Human Locomotion via Feature-Wise Transformations and Local Motion Phasescited by 71
  3. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 67
  4. 2022Weakly Supervised Representation Learning with Sparse Perturbationscited by 64
  5. 2023Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcutscited by 63
  6. 2023Compositional Generalization from First Principlescited by 62
  7. 2023Concept Algebra for (Score-Based) Text-Controlled Generative Modelscited by 59
  8. 2023Identifiability Results for Multimodal Contrastive Learningcited by 58
  9. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 58
  10. 2023Causal Component Analysiscited by 56
  11. 2023Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolationcited by 56
  12. 2021Identifiable Deep Generative Models via Sparse Decodingcited by 54
  13. 2022Causal Discovery in Heterogeneous Environments Under the Sparse Mechanism Shift Hypothesiscited by 54
  14. 2022Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systemscited by 50
  15. 2022Mechanistic Mode Connectivitycited by 50
  16. 2023Score-based Causal Representation Learning with Interventionscited by 49
  17. 2023Causal normalizing flows: from theory to practicecited by 49
  18. 2021Learning Disentangled Representations in the Imaging Domaincited by 46
  19. 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
  20. 2023Provably Learning Object-Centric Representationscited by 45
  21. 2022Neural Manifold Clustering and Embeddingcited by 44
  22. 2022Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learningcited by 44
  23. 2022Harnessing Out-Of-Distribution Examples via Augmenting Content and Stylecited by 43
  24. 2023Causal Discovery from Temporal Data: An Overview and New Perspectivescited by 43
  25. 2023Self-supervised learning of Split Invariant Equivariant representationscited by 41
  26. 2023Provable Compositional Generalization for Object-Centric Learningcited by 41
  27. 2022From Statistical to Causal Learningcited by 40
  28. 2022Do More Negative Samples Necessarily Hurt in Contrastive Learning?cited by 39
  29. 2023Understanding Masked Autoencoders via Hierarchical Latent Variable Modelscited by 39
  30. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 39
  31. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  32. 2021Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspectivecited by 37
  33. 2021On the Limitations of Multimodal VAEscited by 37
  34. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  35. 2023Beyond Normal: On the Evaluation of Mutual Information Estimatorscited by 37
  36. 2022Understanding and Improving the Role of Projection Head in Self-Supervised Learningcited by 36
  37. 2023Measuring axiomatic soundness of counterfactual image modelscited by 34
  38. 2023General Identifiability and Achievability for Causal Representation Learningcited by 34

50 most cited of 308 All 308 (JSON)