mlsanityData to 7 Oct 2026

Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere

arXiv 2005.10242 ↗First posted 20 May 2020cs.LG cs.CV stat.MLcited by 1,393

Not on the map. Why?

Cited by 46 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive representation learning has been outstandingly successful in practice. In this work, we identify two key properties related to the contrastive loss: (1) alignment (closeness) of features from positive pairs, and (2) uniformity of the induced distribution of the (normalized) features on the hypersphere. We prove that, asymptotically, the contrastive loss optimizes these properties, and analyze their positive effects on downstream tasks. Empirically, we introduce an optimizable metric to quantify each property. Extensive experiments on standard vision and language datasets confirm the strong agreement between both metrics and downstream task performance. Remarkably, directly optimizing for these two metrics leads to representations with comparable or better performance at downstream tasks than contrastive learning. Project Page: https://tongzhouwang.info/hypersphere Code: https://github.com/SsnL/align_uniform , https://github.com/SsnL/moco_align_uniform

Key facts. Cited by 1,393 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 148; 2022: 270; 2023: 269; 2024: 235; 2025: 220; 2026: 251.

1,393 papers.

Sort:

By area:

  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  3. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
  4. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  5. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  6. 2024The Platonic Representation Hypothesiscited by 362
  7. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  8. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  9. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  10. 2023A Cookbook of Self-Supervised Learningcited by 248
  11. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  12. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
38 more
  1. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  2. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  3. 2021Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Datacited by 214
  4. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  5. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  6. 2021Unsupervised Semantic Segmentation by Contrasting Object Mask Proposalscited by 190
  7. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  8. 2022SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentationcited by 171
  9. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  10. 2023TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Seriescited by 165
  11. 2021A Unified Objective for Novel Class Discoverycited by 161
  12. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  13. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  14. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 150
  15. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  16. 2022Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphscited by 144
  17. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 144
  18. 2021Decoupled Contrastive Learningcited by 141
  19. 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 138
  20. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  21. 2021Targeted Supervised Contrastive Learning for Long-Tailed Recognitioncited by 132
  22. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  23. 2022How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?cited by 128
  24. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learnerscited by 128
  25. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  26. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  27. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  28. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  29. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 117
  30. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  31. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  32. 2021An Empirical Study of Graph Contrastive Learningcited by 113
  33. 2022Learnable latent embeddings for joint behavioral and neural analysiscited by 113
  34. 2023SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modelingcited by 112
  35. 2022Hyperbolic Vision Transformers: Combining Improvements in Metric Learningcited by 109
  36. 2021Efficient Training of Visual Transformers with Small Datasetscited by 107
  37. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  38. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105

50 most cited of 1,393 All 1,393 (JSON)