# DebCSE: Rethinking Unsupervised Contrastive Sentence Embedding Learning in the Debiasing Perspective

arXiv: https://arxiv.org/abs/2309.07396 · HTML: https://mlsanity.com/paper/2309.07396/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2204.10298/) (2022)
- [An Unsupervised Sentence Embedding Method by Mutual Information Maximization](https://mlsanity.com/paper/2009.12061/) (2020)
- [On the Sentence Embeddings from Pre-trained Language Models](https://mlsanity.com/paper/2011.05864/) (2020)
- [PromptBERT: Improving BERT Sentence Embeddings with Prompts](https://mlsanity.com/paper/2201.04337/) (2022)
- [ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding](https://mlsanity.com/paper/2109.04380/) (2021)
- [Unbiased Learning to Rank with Unbiased Propensity Estimation](https://mlsanity.com/paper/1804.05938/) (2018)
- [Debiased Contrastive Learning of Unsupervised Sentence Representations](https://mlsanity.com/paper/2205.00656/) (2022)
- [SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples](https://mlsanity.com/paper/2201.05979/) (2022)
- [Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives](https://mlsanity.com/paper/2206.02457/) (2022)

## Built on by (most cited first, 5 of 5)

- [VisionTrap: Vision-Augmented Trajectory Prediction Guided by Textual Descriptions](https://mlsanity.com/paper/2407.12345/) (2024)
- [Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning](https://mlsanity.com/paper/2409.12887/) (2024)
- [DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning](https://mlsanity.com/paper/2502.00305/) (2025)
- [Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models](https://mlsanity.com/paper/2502.13656/) (2025)
- [IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressions](https://mlsanity.com/paper/2605.22247/) (2026)

## Abstract (arXiv)

Several prior studies have suggested that word frequency biases can cause the Bert model to learn indistinguishable sentence embeddings. Contrastive learning schemes such as SimCSE and ConSERT have already been adopted successfully in unsupervised sentence embedding to improve the quality of embeddings by reducing this bias. However, these methods still introduce new biases such as sentence length bias and false negative sample bias, that hinders model's ability to learn more fine-grained semantics. In this paper, we reexamine the challenges of contrastive sentence embedding learning from a debiasing perspective and argue that effectively eliminating the influence of various biases is crucial for learning high-quality sentence embeddings. We think all those biases are introduced by simple rules for constructing training data in contrastive learning and the key for contrastive learning sentence embedding is to mimic the distribution of training data in supervised machine learning in unsupervised way. We propose a novel contrastive framework for sentence embedding, termed DebCSE, which can eliminate the impact of these biases by an inverse propensity weighted sampling method to select high-quality positive and negative pairs according to both the surface and semantic similarity between sentences. Extensive experiments on semantic textual similarity (STS) benchmarks reveal that DebCSE significantly outperforms the latest state-of-the-art models with an average Spearman's correlation coefficient of 80.33% on BERTbase.