mlsanityData to 7 Oct 2026

CLEAR: Contrastive Learning for Sentence Representation

arXiv 2012.15466 ↗First posted 31 Dec 2020cs.CLcited by 186

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models have proven their unique powers in capturing implicit language features. However, most pre-training approaches focus on the word-level training objective, while sentence-level objectives are rarely studied. In this paper, we propose Contrastive LEArning for sentence Representation (CLEAR), which employs multiple sentence-level augmentation strategies in order to learn a noise-invariant sentence representation. These augmentations include word and span deletion, reordering, and substitution. Furthermore, we investigate the key reasons that make contrastive learning effective through numerous experiments. We observe that different sentence augmentations during pre-training lead to different performance improvements on various downstream tasks. Our approach is shown to outperform multiple existing methods on both SentEval and GLUE benchmarks.

Key facts. Cited by 186 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 49; 2022: 58; 2023: 38; 2024: 26; 2025: 13; 2026: 2.

186 papers.

Sort:

By area:

  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  3. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  4. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  5. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  6. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  7. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  8. 2021Generating Datasets with Pretrained Language Modelscited by 192
  9. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  10. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  11. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  12. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
38 more
  1. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  2. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  3. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  4. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  5. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  6. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  7. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  8. 2022Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddingscited by 75
  9. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 73
  10. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  11. 2021Contrastive Self-supervised Sequential Recommendation with Robust Augmentationcited by 56
  12. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  13. 2021High-Dimensional Bayesian Optimisation with Variational Autoencoders and Deep Metric Learningcited by 54
  14. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 54
  15. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53
  16. 2021A Survey on Green Deep Learningcited by 52
  17. 2022Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classificationcited by 52
  18. 2021CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understandingcited by 50
  19. 2021Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuningcited by 46
  20. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
  21. 2022Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Codingcited by 40
  22. 2021Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answeringcited by 39
  23. 2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
  24. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  25. 2022UCTopic: Unsupervised Contrastive Learning for Phrase Representations and Topic Miningcited by 33
  26. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  27. 2021TaCL: Improving BERT Pre-training with Token-aware Contrastive Learningcited by 32
  28. 2021Pairwise Supervised Contrastive Learning of Sentence Representationscited by 31
  29. 2022PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddingscited by 29
  30. 2023ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Triggercited by 29
  31. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  32. 2023GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphscited by 28
  33. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  34. 2022Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embeddingcited by 25
  35. 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 25
  36. 2022C2-CRS: Coarse-to-Fine Contrastive Learning for Conversational Recommender Systemcited by 24
  37. 2022LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrievalcited by 24
  38. 2025Retrieval Augmented Generation with Collaborative Filtering for Personalized Text Generationcited by 23

50 most cited of 186 All 186 (JSON)