mlsanityData to 7 Oct 2026

Clustering-Aware Negative Sampling for Unsupervised Sentence Representation

arXiv 2305.09892 ↗First posted 17 May 2023cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Contrastive learning has been widely studied in sentence representation learning. However, earlier works mainly focus on the construction of positive examples, while in-batch samples are often simply treated as negative examples. This approach overlooks the importance of selecting appropriate negative examples, potentially leading to a scarcity of hard negatives and the inclusion of false negatives. To address these issues, we propose ClusterNS (Clustering-aware Negative Sampling), a novel method that incorporates cluster information into contrastive learning for unsupervised sentence representation learning. We apply a modified K-means clustering algorithm to supply hard negatives and recognize in-batch false negatives during training, aiming to solve the two issues in one unified framework. Experiments on semantic textual similarity (STS) tasks demonstrate that our proposed ClusterNS compares favorably with baselines in unsupervised sentence representation learning. Our code has been made publicly available.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 38 of its 73 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

38 of 73 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  9. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  11. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  12. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
26 more
  1. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  2. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  3. 2020Debiased Contrastive Learningcited by 354
  4. 2015Skip-Thought Vectorscited by 321
  5. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  6. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  7. 2016Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clusteringcited by 237
  8. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  9. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  10. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  11. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  12. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  13. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 90
  14. 2021Supporting Clustering with Contrastive Learningcited by 90
  15. 2020Hard negative examples are hard, but usefulcited by 82
  16. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  17. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 73
  18. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  19. 2022Prototypical Verbalizer for Prompt-based Few-shot Tuningcited by 53
  20. 2022SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samplescited by 39
  21. 2017Self-Taught Convolutional Neural Networks for Short Text Clusteringcited by 34
  22. 2021Prototypical Representation Learning for Relation Extractioncited by 33
  23. 2021Pairwise Supervised Contrastive Learning of Sentence Representationscited by 31
  24. 2015Text Understanding from Scratchcited by 25
  25. 2022Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negativescited by 16
  26. 2022Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embeddingcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 4.

5 papers.

  1. 2023A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyondcited by 11
  2. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  3. 2025DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learningcited by 2
  4. 2025Cross-Modal Clustering-Guided Negative Sampling for Self-Supervised Joint Learning from Medical Images and Reportscited by 0
  5. 2025Text2Token: Unsupervised Text Representation Learning with Token Target Predictioncited by 0