mlsanityData to 14 Oct 2026

Diversity Is All You Need for Contrastive Learning: Spectral Bounds on Gradient Magnitudes

arXiv 2510.05767 ↗First posted 7 Oct 2025cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

We derive non-asymptotic spectral bands that bound the squared InfoNCE gradient norm via alignment, temperature, and batch spectrum, recovering the (1/τ2) law and closely tracking batch-mean gradients on synthetic data and ImageNet. Using effective rank (Reff) as an anisotropy proxy, we design spectrum-aware batch selection, including a fast greedy builder. On ImageNet-100, Greedy-64 cuts time-to-67.5% top-1 by 15% vs. random (24% vs. Pool--P3) at equal accuracy; CIFAR-10 shows similar gains. In-batch whitening promotes isotropy and reduces 50-step gradient variance by (1.37×), matching our theoretical upper bound.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 18 of its 28 references that are in the map. First posted on arXiv 7 Oct 2025.

Builds on

18 of 28 references.

References · 18

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  6. 2020Supervised Contrastive Learningcited by 2,829
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  9. 2020Contrastive Learning with Hard Negative Samplescited by 628
  10. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  11. 2015An Introduction to Matrix Concentration Inequalitiescited by 397
  12. 2020Debiased Contrastive Learningcited by 356
6 more
  1. 2017Sampling Matters in Deep Embedding Learningcited by 285
  2. 2020Whitening for Self-Supervised Representation Learningcited by 279
  3. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  4. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  5. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  6. 2017Smart Mining for Deep Metric Learningcited by 96

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Variance-reduced Domain Adaptation using Paired Samplingcited by 0
  2. 2026Online Variance Reduction for Domain Adaptation on Streaming Datacited by 0