# SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives

arXiv: https://arxiv.org/abs/2305.13192 · HTML: https://mlsanity.com/paper/2305.13192/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 59 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 30 of 37)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [CERT: Contrastive Self-supervised Learning for Language Understanding](https://mlsanity.com/paper/2005.12766/) (2020)
- [DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2204.10298/) (2022)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](https://mlsanity.com/paper/1703.02507/) (2017)
- [Self-Guided Contrastive Learning for BERT Sentence Representations](https://mlsanity.com/paper/2106.07345/) (2021)
- [Learning Distributed Representations of Sentences from Unlabelled Data](https://mlsanity.com/paper/1602.03483/) (2016)
- [An Unsupervised Sentence Embedding Method by Mutual Information Maximization](https://mlsanity.com/paper/2009.12061/) (2020)
- [RobBERT: a Dutch RoBERTa-based Language Model](https://mlsanity.com/paper/2001.06286/) (2020)
- [ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning](https://mlsanity.com/paper/2012.15022/) (2020)
- [Equivariant Contrastive Learning](https://mlsanity.com/paper/2111.00899/) (2021)

## Built on by (most cited first, 15 of 15)

- [Enhancing Idiomatic Representation in Multiple Languages via an Adaptive Contrastive Triplet Loss](https://mlsanity.com/paper/2406.15175/) (2024)
- [Evaluating the Effect of Retrieval Augmentation on Social Biases](https://mlsanity.com/paper/2502.17611/) (2025)
- [MAPGD: Multi-Agent Prompt Gradient Descent for Collaborative Prompt Optimization](https://mlsanity.com/paper/2509.11361/) (2025)
- [DistillCSE: Distilled Contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2310.13499/) (2023)
- [Instruction Embedding: Latent Representations of Instructions Towards Task Identification](https://mlsanity.com/paper/2409.19680/) (2024)
- [TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2401.06620/) (2024)
- [Non-contrastive sentence representations via self-supervision](https://mlsanity.com/paper/2310.17690/) (2023)
- [Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learning](https://mlsanity.com/paper/2408.06043/) (2024)
- [Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers](https://mlsanity.com/paper/2512.10422/) (2025)
- [CASE -- Condition-Aware Sentence Embeddings for Conditional Semantic Textual Similarity Measurement](https://mlsanity.com/paper/2503.17279/) (2025)
- [Towards Better Understanding of Contrastive Sentence Representation Learning: A Unified Paradigm for Gradient](https://mlsanity.com/paper/2402.18281/) (2024)
- [Unveiling Key Aspects of Fine-Tuning in Sentence Embeddings: A Representation Rank Analysis](https://mlsanity.com/paper/2405.11297/) (2024)
- [Understanding InfoNCE: Transition Probability Matrix Induced Feature Clustering](https://mlsanity.com/paper/2511.12180/) (2025)
- [Map of Encoders -- Mapping Sentence Encoders using Quantum Relative Entropy](https://mlsanity.com/paper/2602.08740/) (2026)
- [G-Loss: Graph-Guided Fine-Tuning of Language Models](https://mlsanity.com/paper/2604.25853/) (2026)

## Abstract (arXiv)

This paper improves contrastive learning for sentence embeddings from two perspectives: handling dropout noise and addressing feature corruption. Specifically, for the first perspective, we identify that the dropout noise from negative pairs affects the model's performance. Therefore, we propose a simple yet effective method to deal with such type of noise. Secondly, we pinpoint the rank bottleneck of current solutions to feature corruption and propose a dimension-wise contrastive learning objective to address this issue. Both proposed methods are generic and can be applied to any contrastive learning based models for sentence embeddings. Experimental results on standard benchmarks demonstrate that combining both proposed methods leads to a gain of 1.8 points compared to the strong baseline SimCSE configured with BERT base. Furthermore, applying the proposed method to DiffCSE, another strong contrastive learning based baseline, results in a gain of 1.4 points.