mlsanityData to 14 Oct 2026

DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings

arXiv 2204.10298 ↗First posted 21 Apr 2022cs.CLcited by 132

Not on the map. Why?

Abstract (arXiv)

We propose DiffCSE, an unsupervised contrastive learning framework for learning sentence embeddings. DiffCSE learns sentence embeddings that are sensitive to the difference between the original sentence and an edited sentence, where the edited sentence is obtained by stochastically masking out the original sentence and then sampling from a masked language model. We show that DiffSCE is an instance of equivariant contrastive learning (Dangovski et al., 2021), which generalizes contrastive learning and learns representations that are insensitive to certain types of augmentations and sensitive to other "harmful" types of augmentations. Our experiments show that DiffCSE achieves state-of-the-art results among unsupervised sentence representation learning methods, outperforming unsupervised SimCSE by 2.3 absolute points on semantic textual similarity tasks.

Key facts. Cited by 132 papers in the map (data through 14 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 21 Apr 2022.

Builds on

26 of 41 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  11. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  12. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
14 more
  1. 2015Skip-Thought Vectorscited by 322
  2. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  3. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  4. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  5. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 202
  6. 2018An efficient framework for learning sentence representationscited by 198
  7. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  8. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  9. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  10. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 91
  11. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  12. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  13. 2021Equivariant Contrastive Learningcited by 66
  14. 2020Universal Sentence Representation Learning with Conditional Masked Language Modelcited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 18; 2023: 57; 2024: 32; 2025: 15; 2026: 10.

132 papers.

Sort:

By area:

  1. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  2. 2023AnglE-optimized Text Embeddingscited by 99
  3. 2023Scaling Sentence Embeddings with Large Language Modelscited by 85
  4. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  5. 2023RaSa: Relation and Sensitivity Aware Representation Learning for Text-based Person Searchcited by 54
  6. 2023BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddingscited by 32
  7. 2023Improving Contrastive Learning of Sentence Embeddings from AI Feedbackcited by 29
  8. 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 26
  9. 2023RankCSE: Unsupervised Sentence Representations Learning via Learning to Rankcited by 23
  10. 2022UnifieR: A Unified Retriever for Large-Scale Retrievalcited by 19
  11. 2022ContraCLM: Contrastive Learning For Causal Language Modelcited by 18
  12. 2024Question-Based Retrieval using Atomic Units for Enterprise RAGcited by 17
38 more
  1. 2024Meta-Task Prompting Elicits Embeddings from Large Language Modelscited by 16
  2. 2022Ranking-Enhanced Unsupervised Sentence Representation Learningcited by 15
  3. 2023SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectivescited by 15
  4. 2024BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectivescited by 15
  5. 2023Composition-contrastive Learning for Sentence Embeddingscited by 14
  6. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 13
  7. 2022Don't Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Poolingcited by 12
  8. 2023A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyondcited by 12
  9. 2023CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verificationcited by 11
  10. 2023ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systemscited by 11
  11. 2022Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddingscited by 10
  12. 2023TimeDRL: Disentangled Representation Learning for Multivariate Time-Seriescited by 10
  13. 2022Differentiable Data Augmentation for Contrastive Sentence Representation Learningcited by 9
  14. 2024Understanding and Mitigating the Threat of Vec2Text to Dense Retrieval Systemscited by 9
  15. 2024Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Modelscited by 9
  16. 2024D2LLM: Decomposed and Distilled Large Language Models for Semantic Searchcited by 9
  17. 2022SDA: Simple Discrete Augmentation for Contrastive Sentence Representation Learningcited by 8
  18. 2023CoT-BERT: Enhancing Unsupervised Sentence Representation through Chain-of-Thoughtcited by 8
  19. 20242D Matryoshka Sentence Embeddingscited by 8
  20. 2024Hyper-CL: Conditioning Sentence Representations with Hypernetworkscited by 8
  21. 2022miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddingscited by 7
  22. 2023Logic Against Bias: Textual Entailment Mitigates Stereotypical Sentence Reasoningcited by 7
  23. 2023Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCEcited by 7
  24. 2023Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networkscited by 6
  25. 2023Efficient Domain Adaptation of Sentence Embeddings Using Adapterscited by 6
  26. 2023Length is a Curse and a Blessing for Document-level Semanticscited by 6
  27. 2023How Well Do Text Embedding Models Understand Syntax?cited by 6
  28. 2023Improving Self-supervised Molecular Representation Learning using Persistent Homologycited by 6
  29. 2024Fine-tuning CLIP Text Encoders with Two-step Paraphrasingcited by 6
  30. 2022Masked Autoencoders As The Unified Learners For Pre-Trained Sentence Representationcited by 5
  31. 2022AugCSE: Contrastive Sentence Embedding with Diverse Augmentationscited by 5
  32. 2023Clustering-Aware Negative Sampling for Unsupervised Sentence Representationcited by 5
  33. 2023Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operationscited by 5
  34. 2023Revealing the Blind Spot of Sentence Encoder Evaluation by HEROScited by 5
  35. 2023AspectCSE: Sentence Embeddings for Aspect-based Semantic Textual Similarity Using Contrastive Learning and Structured Knowledgecited by 5
  36. 2023DebCSE: Rethinking Unsupervised Contrastive Sentence Embedding Learning in the Debiasing Perspectivecited by 5
  37. 2023On the Dimensionality of Sentence Embeddingscited by 5
  38. 2023Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Wordscited by 5

50 most cited of 132 All 132 (JSON)