mlsanityData to 7 Oct 2026

SimCSE: Simple Contrastive Learning of Sentence Embeddings

arXiv 2104.08821 ↗First posted 18 Apr 2021cs.CL cs.LGcited by 2,168

Not on the map. Why?

Cited by 77 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents SimCSE, a simple contrastive learning framework that greatly advances state-of-the-art sentence embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previous supervised counterparts. We find that dropout acts as minimal data augmentation, and removing it leads to a representation collapse. Then, we propose a supervised approach, which incorporates annotated pairs from natural language inference datasets into our contrastive learning framework by using "entailment" pairs as positives and "contradiction" pairs as hard negatives. We evaluate SimCSE on standard semantic textual similarity (STS) tasks, and our unsupervised and supervised models using BERT base achieve an average of 76.3% and 81.6% Spearman's correlation respectively, a 4.2% and 2.2% improvement compared to the previous best results. We also show -- both theoretically and empirically -- that the contrastive learning objective regularizes pre-trained embeddings' anisotropic space to be more uniform, and it better aligns positive pairs when supervised signals are available.

Key facts. Cited by 2,168 papers in the map (data through 7 Oct 2026). Builds on 37 of its 68 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

37 of 68 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
25 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  2. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  3. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  4. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  5. 2015Skip-Thought Vectorscited by 321
  6. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  7. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  8. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  9. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  10. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 228
  11. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  12. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  13. 2018An efficient framework for learning sentence representationscited by 197
  14. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  15. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  16. 2019Learning Dense Representations for Entity Retrievalcited by 122
  17. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 103
  18. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  19. 2020Learning Dense Representations of Phrases at Scalecited by 91
  20. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 90
  21. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  22. 2017On Sampling Strategies for Neural Network-based Collaborative Filteringcited by 71
  23. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 68
  24. 2017ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translationscited by 44
  25. 2019A Bilingual Generative Transformer for Semantic Sentence Embeddingcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 89; 2022: 406; 2023: 497; 2024: 508; 2025: 384; 2026: 284.

2,168 papers.

Sort:

By area:

  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  3. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  4. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  5. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  6. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  7. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  8. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  9. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  10. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  11. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 463
  12. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
38 more
  1. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  2. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  3. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  4. 2024The Platonic Representation Hypothesiscited by 362
  5. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  6. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
  7. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  8. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  9. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  10. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  11. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  12. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  13. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  14. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
  15. 2023Improving Text Embeddings with Large Language Modelscited by 295
  16. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  17. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  18. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  19. 2023A Cookbook of Self-Supervised Learningcited by 248
  20. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  21. 2023Do Large Language Models Know What They Don't Know?cited by 238
  22. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  23. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  24. 2024Generative Representational Instruction Tuningcited by 227
  25. 2022A Contrastive Framework for Neural Text Generationcited by 224
  26. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  27. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  28. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  29. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  30. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  31. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  32. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  33. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  34. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  35. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  36. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  37. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 158
  38. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158

50 most cited of 2,168 All 2,168 (JSON)