# Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learning

arXiv: https://arxiv.org/abs/2111.14585 · HTML: https://mlsanity.com/paper/2111.14585/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 68 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Builds on (most cited first, 30 of 51)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)

## Built on by (most cited first, 18 of 18)

- [Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability](https://mlsanity.com/paper/2411.19943/) (2024)
- [IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-training](https://mlsanity.com/paper/2310.07355/) (2023)
- [COMEDIAN: Self-Supervised Learning and Knowledge Distillation for Action Spotting using Transformers](https://mlsanity.com/paper/2309.01270/) (2023)
- [mathbbX-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs](https://mlsanity.com/paper/2407.18134/) (2024)
- [Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding](https://mlsanity.com/paper/2205.00693/) (2022)
- [CoUn: Empowering Machine Unlearning via Contrastive Learning](https://mlsanity.com/paper/2509.16391/) (2025)
- [Proposal-Contrastive Pretraining for Object Detection from Fewer Data](https://mlsanity.com/paper/2310.16835/) (2023)
- [Open-Vocabulary Mobile Manipulation Based on Double Relaxed Contrastive Learning with Dense Labeling](https://mlsanity.com/paper/2412.16576/) (2024)
- [A Large-Scale Analysis on Contextual Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2504.06153/) (2025)
- [CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillation](https://mlsanity.com/paper/2504.21478/) (2025)
- [From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport](https://mlsanity.com/paper/2310.11439/) (2023)
- [MNN: Mixed Nearest-Neighbors for Self-Supervised Learning](https://mlsanity.com/paper/2311.00562/) (2023)
- [Enhancing Fine-grained Object Detection in Aerial Images via Orthogonal Mapping](https://mlsanity.com/paper/2407.17738/) (2024)
- [Dual Advancement of Representation Learning and Clustering for Sparse and Noisy Images](https://mlsanity.com/paper/2409.01781/) (2024)
- [MSVQ: Self-Supervised Learning with Multiple Sample Views and Queues](https://mlsanity.com/paper/2305.05370/) (2023)
- [KDC-MAE: Knowledge Distilled Contrastive Mask Auto-Encoder](https://mlsanity.com/paper/2411.12270/) (2024)
- [Predicting butterfly species presence from satellite imagery using soft contrastive regularisation](https://mlsanity.com/paper/2505.09306/) (2025)
- [SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization](https://mlsanity.com/paper/2605.08809/) (2026)

## Abstract (arXiv)

Contrastive representation learning has proven to be an effective self-supervised learning method. Most successful approaches are based on Noise Contrastive Estimation (NCE) and use different views of an instance as positives that should be contrasted with other instances, called negatives, that are considered as noise. However, several instances in a dataset are drawn from the same distribution and share underlying semantic information. A good data representation should contain relations, or semantic similarity, between the instances. Contrastive learning implicitly learns relations but considering all negatives as noise harms the quality of the learned relations. To circumvent this issue, we propose a novel formulation of contrastive learning using semantic similarity between instances called Similarity Contrastive Estimation (SCE). Our training objective is a soft contrastive learning one. Instead of hard classifying positives and negatives, we estimate from one view of a batch a continuous distribution to push or pull instances based on their semantic similarities. This target similarity distribution is sharpened to eliminate noisy relations. The model predicts for each instance, from another view, the target distribution while contrasting its positive with negatives. Experimental results show that SCE is Top-1 on the ImageNet linear evaluation protocol at 100 pretraining epochs with 72.1% accuracy and is competitive with state-of-the-art algorithms by reaching 75.4% for 200 epochs with multi-crop. We also show that SCE is able to generalize to several tasks. Source code is available here: https://github.com/CEA-LIST/SCE.