mlsanityData to 7 Oct 2026

Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses

arXiv 2405.18045 ↗First posted 28 May 2024cs.LG cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

What do different contrastive learning (CL) losses actually optimize for? Although multiple CL methods have demonstrated remarkable representation learning capabilities, the differences in their inner workings remain largely opaque. In this work, we analyse several CL families and prove that, under certain conditions, they admit the same minimisers when optimizing either their batch-level objectives or their expectations asymptotically. In both cases, an intimate connection with the hyperspherical energy minimisation (HEM) problem resurfaces. Drawing inspiration from this, we introduce a novel CL objective, coined Decoupled Hyperspherical Energy Loss (DHEL). DHEL simplifies the problem by decoupling the target hyperspherical energy from the alignment of positive examples while preserving the same theoretical guarantees. Going one step further, we show the same results hold for another relevant CL family, namely kernel contrastive learning (KCL), with the additional advantage of the expected loss being independent of batch size, thus identifying the minimisers in the non-asymptotic regime. Empirical results demonstrate improved downstream performance and robustness across combinations of different batch sizes and hyperparameters and reduced dimensionality collapse, on several computer vision datasets.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

37 of 52 references.

References · 37

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  5. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  6. 2019Contrastive Multiview Codingcited by 1,297
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  8. 2020Contrastive Learning with Hard Negative Samplescited by 619
  9. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  10. 2020Understanding the Behaviour of Contrastive Losscited by 397
  11. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  12. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
25 more
  1. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  2. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  3. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  4. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  5. 2021Decoupled Contrastive Learningcited by 141
  6. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  7. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  8. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  9. 2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
  10. 2018Learning towards Minimum Hyperspherical Energycited by 75
  11. 2022How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learningcited by 65
  12. 2021Dissecting Supervised Contrastive Learningcited by 64
  13. 2021The Power of Contrast for Feature Learning: A Theoretical Analysiscited by 53
  14. 2022Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCocited by 52
  15. 2020Unsupervised Feature Learning by Cross-Level Instance-Group Discriminationcited by 51
  16. 2021Learning with Hyperspherical Uniformitycited by 35
  17. 2022Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functionscited by 32
  18. 2019Regularizing Neural Networks via Minimizing Hyperspherical Energycited by 25
  19. 2023Structuring Representation Geometry with Rotationally Equivariant Contrastive Learningcited by 24
  20. 2022Conditional Contrastive Learning with Kernelcited by 22
  21. 2023Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gapcited by 14
  22. 2022Joint Embedding Self-Supervised Learning in the Kernel Regimecited by 12
  23. 2022Integrating Prior Knowledge in Contrastive Learning with Kernelcited by 11
  24. 2023Mini-Batch Optimization of Contrastive Losscited by 8
  25. 2023Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysiscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025Multimodal Reasoning Agent for Zero-Shot Composed Image Retrievalcited by 7
  2. 2025A Principled Framework for Multi-View Contrastive Learningcited by 4
  3. 2025SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularizationcited by 3
  4. 2025On the Similarities of Embeddings in Contrastive Learningcited by 2
  5. 2025Understanding InfoNCE: Transition Probability Matrix Induced Feature Clusteringcited by 0
  6. 2026Neural Collapse by Design: Learning Class Prototypes on the Hyperspherecited by 0