# Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCE

arXiv: https://arxiv.org/abs/2310.06918 · HTML: https://mlsanity.com/paper/2310.06918/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Circle Loss: A Unified Perspective of Pair Similarity Optimization](https://mlsanity.com/paper/2002.10857/) (2020)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2204.10298/) (2022)
- [ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding](https://mlsanity.com/paper/2109.04380/) (2021)
- [Debiased Contrastive Learning of Unsupervised Sentence Representations](https://mlsanity.com/paper/2205.00656/) (2022)
- [Smoothed Contrastive Learning for Unsupervised Sentence Embedding](https://mlsanity.com/paper/2109.04321/) (2021)
- [Improving Adversarial Robustness with Self-Paced Hard-Class Pair Reweighting](https://mlsanity.com/paper/2210.15068/) (2022)

## Built on by (most cited first, 7 of 7)

- [LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning](https://mlsanity.com/paper/2503.04812/) (2025)
- [U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs](https://mlsanity.com/paper/2507.14902/) (2025)
- [Efficient Self-Supervised Learning for Earth Observation via Dynamic Dataset Curation](https://mlsanity.com/paper/2504.06962/) (2025)
- [Advancing Few-Shot Pediatric Arrhythmia Classification with a Novel Contrastive Loss and Multimodal Learning](https://mlsanity.com/paper/2509.19315/) (2025)
- [PACE: Progressive Angular-to-Norm Contrastive Embedding](https://mlsanity.com/paper/2609.15152/) (2026)
- [Harnessing Attention Mechanisms: Efficient Sequence Reduction using Attention-based Autoencoders](https://mlsanity.com/paper/2310.14837/) (2023)
- [Combating Visual Neglect and Semantic Drift in Large Multimodal Models for Enhanced Cross-Modal Retrieval](https://mlsanity.com/paper/2604.25273/) (2026)

## Abstract (arXiv)

The recent success of SimCSE has greatly advanced state-of-the-art sentence representations. However, the original formulation of SimCSE does not fully exploit the potential of hard negative samples in contrastive learning. This study introduces an unsupervised contrastive learning framework that combines SimCSE with hard negative mining, aiming to enhance the quality of sentence embeddings. The proposed focal-InfoNCE function introduces self-paced modulation terms in the contrastive objective, downweighting the loss associated with easy negatives and encouraging the model focusing on hard negatives. Experimentation on various STS benchmarks shows that our method improves sentence embeddings in terms of Spearman's correlation and representation alignment and uniformity.