mlsanityData to 7 Oct 2026

The Hidden Pitfalls of the Cosine Similarity Loss

arXiv 2406.16468 ↗First posted 24 Jun 2024cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

We show that the gradient of the cosine similarity between two points goes to zero in two under-explored settings: (1) if a point has large magnitude or (2) if the points are on opposite ends of the latent space. Counterintuitively, we prove that optimizing the cosine similarity between points forces them to grow in magnitude. Thus, (1) is unavoidable in practice. We then observe that these derivations are extremely general -- they hold across deep learning architectures and for many of the standard self-supervised learning (SSL) loss functions. This leads us to propose cut-initialization: a simple change to network initialization that helps all studied SSL methods converge faster.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 24 of its 28 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

24 of 28 references.

References · 24

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  10. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  11. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  12. 2021Deep Long-Tailed Learning: A Surveycited by 416
12 more
  1. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  2. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  3. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  4. 2017NormFace: L2 Hypersphere Embedding for Face Verificationcited by 237
  5. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  6. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  7. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  8. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  9. 2020BYOL works even without batch statisticscited by 95
  10. 2022How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learningcited by 65
  11. 2020Deep Metric Learning with Spherical Embeddingcited by 25
  12. 2023The Edge of Orthogonality: A Simple View of What Makes BYOL Tickcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  2. 2025On the Importance of Embedding Norms in Self-Supervised Learningcited by 9
  3. 2025On the Role of Weight Decay in Collaborative Filtering: A Popularity Perspectivecited by 2
  4. 2025Explainable AI for Analyzing Person-Specific Patterns in Facial Recognition Taskscited by 1
  5. 2026Private and Robust Contribution Evaluation in Federated Learningcited by 1
  6. 2026RXNRECer Enables Fine-grained Enzymatic Function Annotation through Active Learning and Protein Language Modelscited by 0
  7. 2026HQ-JEPA: Hybrid Quantum Joint-Embedding Predictive Architecture for Cross-Modal Remote Sensing Representation Learningcited by 0