mlsanityData to 14 Oct 2026

Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning

arXiv 2206.01342 ↗First posted 2 Jun 2022cs.LG cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

While the empirical success of self-supervised learning (SSL) heavily relies on the usage of deep nonlinear models, existing theoretical works on SSL understanding still focus on linear ones. In this paper, we study the role of nonlinearity in the training dynamics of contrastive learning (CL) on one and two-layer nonlinear networks with homogeneous activation h(x) = h'(x)x. We have two major theoretical discoveries. First, the presence of nonlinearity can lead to many local optima even in 1-layer setting, each corresponding to certain patterns from the data distribution, while with linear activation, only one major pattern can be learned. This suggests that models with lots of parameters can be regarded as a brute-force way to find these local optima induced by nonlinearity. Second, in the 2-layer case, linear activation is proven not capable of learning specialized weights into diverse patterns, demonstrating the importance of nonlinearity. In addition, for 2-layer setting, we also discover global modulation: those local patterns discriminative from the perspective of global-level patterns are prioritized to learn, further characterizing the learning process. Simulation verifies our theoretical findings.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

36 of 56 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  12. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
24 more
  1. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  2. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  3. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  4. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  5. 2016On the Expressive Power of Deep Neural Networkscited by 306
  6. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  7. 2016Deep Learning without Poor Local Minimacited by 234
  8. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  9. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  10. 2021The Principles of Deep Learning Theorycited by 173
  11. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  12. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 102
  13. 2019One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizerscited by 102
  14. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  15. 2020The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Modelscited by 91
  16. 2019Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLPcited by 73
  17. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  18. 2021The Power of Contrast for Feature Learning: A Theoretical Analysiscited by 53
  19. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  20. 2021Reproducing Kernel Hilbert Space, Mercer's Theorem, Eigenfunctions, Nystr"om Method, and Use of Kernels in Machine Learning: Tutorial and Surveycited by 37
  21. 2022Understanding Deep Contrastive Learning via Coordinate-wise Optimizationcited by 35
  22. 2020Expressivity of Deep Neural Networkscited by 32
  23. 2021Towards Demystifying Representation Learning with Non-contrastive Self-supervisioncited by 23
  24. 2019Luck Matters: Understanding Training Dynamics of Deep ReLU Networkscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 4; 2025: 5; 2026: 3.

19 papers.

Sort:

  1. 2023A Cookbook of Self-Supervised Learningcited by 250
  2. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 102
  3. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  4. 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40
  5. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 40
  6. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  7. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  8. 2025Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokkingcited by 15
  9. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  10. 2024Collapse-Proof Non-Contrastive Self-Supervised Learningcited by 7
  11. 2023Deciphering the Projection Head: Representation Evaluation Self-supervised Learningcited by 5
  12. 2025Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learningcited by 5
7 more
  1. 2026Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learningcited by 3
  2. 2024Understanding Representation Learnability of Nonlinear Self-Supervised Learningcited by 2
  3. 2025On the Alignment Between Supervised and Self-Supervised Contrastive Learningcited by 2
  4. 2024Training Dynamics of Nonlinear Contrastive Learning Model in the High Dimensional Limitcited by 1
  5. 2025Enhancing Contrastive Link Prediction With Edge Balancing Augmentationcited by 0
  6. 2026Model Evolution Under Zeroth-Order Optimization: A Neural Tangent Kernel Perspectivecited by 0
  7. 2026Which Tasks Survive Self-Supervised Learning?cited by 0