mlsanityData to 7 Oct 2026

SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives

arXiv 2305.13192 ↗First posted 22 May 2023cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

This paper improves contrastive learning for sentence embeddings from two perspectives: handling dropout noise and addressing feature corruption. Specifically, for the first perspective, we identify that the dropout noise from negative pairs affects the model's performance. Therefore, we propose a simple yet effective method to deal with such type of noise. Secondly, we pinpoint the rank bottleneck of current solutions to feature corruption and propose a dimension-wise contrastive learning objective to address this issue. Both proposed methods are generic and can be applied to any contrastive learning based models for sentence embeddings. Experimental results on standard benchmarks demonstrate that combining both proposed methods leads to a gain of 1.8 points compared to the strong baseline SimCSE configured with BERT base. Furthermore, applying the proposed method to DiffCSE, another strong contrastive learning based baseline, results in a gain of 1.4 points.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

37 of 59 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  12. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
25 more
  1. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  2. 2020Debiased Contrastive Learningcited by 354
  3. 2015Skip-Thought Vectorscited by 321
  4. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  5. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  6. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  7. 2018An efficient framework for learning sentence representationscited by 197
  8. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  9. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  10. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  11. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  12. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 103
  13. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  14. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  15. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 90
  16. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  17. 2020ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learningcited by 72
  18. 2021Equivariant Contrastive Learningcited by 65
  19. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  20. 2022PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddingscited by 29
  21. 2015Dropout as data augmentationcited by 27
  22. 2022MCSE: Multimodal Contrastive Learning of Sentence Embeddingscited by 22
  23. 2022Just Rank: Rethinking Evaluation with Word and Sentence Similaritiescited by 20
  24. 2022SCD: Self-Contrastive Decorrelation for Sentence Embeddingscited by 19
  25. 2022A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddingscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 5; 2026: 2.

15 papers.

Sort:

By area:

  1. 2024Enhancing Idiomatic Representation in Multiple Languages via an Adaptive Contrastive Triplet Losscited by 7
  2. 2025Evaluating the Effect of Retrieval Augmentation on Social Biasescited by 7
  3. 2025MAPGD: Multi-Agent Prompt Gradient Descent for Collaborative Prompt Optimizationcited by 6
  4. 2023DistillCSE: Distilled Contrastive Learning for Sentence Embeddingscited by 4
  5. 2024Instruction Embedding: Latent Representations of Instructions Towards Task Identificationcited by 4
  6. 2024TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Modelscited by 3
  7. 2023Non-contrastive sentence representations via self-supervisioncited by 2
  8. 2024Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learningcited by 2
  9. 2025Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layerscited by 2
  10. 2025CASE -- Condition-Aware Sentence Embeddings for Conditional Semantic Textual Similarity Measurementcited by 1
  11. 2024Towards Better Understanding of Contrastive Sentence Representation Learning: A Unified Paradigm for Gradientcited by 0
  12. 2024Unveiling Key Aspects of Fine-Tuning in Sentence Embeddings: A Representation Rank Analysiscited by 0
3 more
  1. 2025Understanding InfoNCE: Transition Probability Matrix Induced Feature Clusteringcited by 0
  2. 2026Map of Encoders -- Mapping Sentence Encoders using Quantum Relative Entropycited by 0
  3. 2026G-Loss: Graph-Guided Fine-Tuning of Language Modelscited by 0