mlsanityData to 7 Oct 2026

On the Importance of Embedding Norms in Self-Supervised Learning

arXiv 2502.09252 ↗First posted 13 Feb 2025cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) allows training data representations without a supervised signal and has become an important paradigm in machine learning. Most SSL methods employ the cosine similarity between embedding vectors and hence effectively embed data on a hypersphere. While this seemingly implies that embedding norms cannot play any role in SSL, a few recent works have suggested that embedding norms have properties related to network convergence and confidence. In this paper, we resolve this apparent contradiction and systematically establish the embedding norm's role in SSL training. Using theoretical analysis, simulations, and experiments, we show that embedding norms (i) govern SSL convergence rates and (ii) encode network confidence, with smaller norms corresponding to unexpected samples. Additionally, we show that manipulating embedding norms can have large effects on convergence speed. Our findings demonstrate that SSL embedding norms are integral to understanding and optimizing network behavior.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 45 of its 50 references that are in the map. First posted on arXiv 13 Feb 2025.

Builds on

45 of 50 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  12. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
33 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  2. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2021Multiscale Vision Transformerscited by 896
  5. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  6. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  7. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  8. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  9. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  10. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  11. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  12. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  13. 2023Demystifying CLIP Datacited by 265
  14. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  15. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 253
  16. 2017NormFace: L2 Hypersphere Embedding for Face Verificationcited by 237
  17. 2019Human uncertainty makes classification more robustcited by 231
  18. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  19. 2021Improving Calibration for Long-Tailed Recognitioncited by 212
  20. 2019Transferrable Prototypical Networks for Unsupervised Domain Adaptationcited by 141
  21. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 122
  22. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  23. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  24. 2022How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learningcited by 65
  25. 2021von Mises-Fisher Loss: An Exploration of Embedding Geometries for Supervised Learningcited by 42
  26. 2023Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputscited by 28
  27. 2020Deep Metric Learning with Spherical Embeddingcited by 25
  28. 2022A Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learningcited by 19
  29. 2023URL: A Representation Learning Benchmark for Transferable Uncertainty Estimatescited by 17
  30. 2018Heated-Up Softmax Embeddingcited by 15
  31. 2023The Edge of Orthogonality: A Simple View of What Makes BYOL Tickcited by 15
  32. 2022Bi-directional Contrastive Learning for Domain Adaptive Semantic Segmentationcited by 14
  33. 2024The Hidden Pitfalls of the Cosine Similarity Losscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2026InfoNCE Induces Gaussian Distributioncited by 14
  2. 2025Semantics at an Angle: When Cosine Similarity Works Until It Doesn'tcited by 13
  3. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  4. 2026Epistemic Uncertainty Quantification for Pre-trained VLMs via Riemannian Flow Matchingcited by 2
  5. 2026Improving Relative Representations with Learned Anchors and Whitened Inner Productscited by 1
  6. 2025CLIP-like Model as a Foundational Density Ratio Estimatorcited by 0
  7. 2026GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embeddingcited by 0
  8. 2026Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Normscited by 0
  9. 2026When Features Become Instances: Inverted Contrastive Learning for Unsupervised Feature Selectioncited by 0