mlsanityData to 14 Oct 2026

Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCE

arXiv 2310.06918 ↗First posted 10 Oct 2023cs.CL cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

The recent success of SimCSE has greatly advanced state-of-the-art sentence representations. However, the original formulation of SimCSE does not fully exploit the potential of hard negative samples in contrastive learning. This study introduces an unsupervised contrastive learning framework that combines SimCSE with hard negative mining, aiming to enhance the quality of sentence embeddings. The proposed focal-InfoNCE function introduces self-paced modulation terms in the contrastive objective, downweighting the loss associated with easy negatives and encouraging the model focusing on hard negatives. Experimentation on various STS benchmarks shows that our method improves sentence embeddings in terms of Spearman's correlation and representation alignment and uniformity.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 20 of its 30 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

20 of 30 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  6. 2020Supervised Contrastive Learningcited by 2,829
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  10. 2020Contrastive Learning with Hard Negative Samplescited by 628
  11. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  12. 2020Circle Loss: A Unified Perspective of Pair Similarity Optimizationcited by 403
8 more
  1. 2020Understanding the Behaviour of Contrastive Losscited by 398
  2. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  3. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  4. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 132
  5. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 74
  6. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  7. 2021Smoothed Contrastive Learning for Unsupervised Sentence Embeddingcited by 15
  8. 2022Improving Adversarial Robustness with Self-Paced Hard-Class Pair Reweightingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 4; 2026: 2.

7 papers.

  1. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 55
  2. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMscited by 10
  3. 2025Efficient Self-Supervised Learning for Earth Observation via Dynamic Dataset Curationcited by 7
  4. 2025Advancing Few-Shot Pediatric Arrhythmia Classification with a Novel Contrastive Loss and Multimodal Learningcited by 1
  5. 2026PACE: Progressive Angular-to-Norm Contrastive Embeddingcited by 1
  6. 2023Harnessing Attention Mechanisms: Efficient Sequence Reduction using Attention-based Autoencoderscited by 0
  7. 2026Combating Visual Neglect and Semantic Drift in Large Multimodal Models for Enhanced Cross-Modal Retrievalcited by 0