# SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples

arXiv: https://arxiv.org/abs/2201.05979 · HTML: https://mlsanity.com/paper/2201.05979/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 16 Jan 2022.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [Intriguing Properties of Contrastive Losses](https://mlsanity.com/paper/2011.02803/) (2020)
- [Can contrastive learning avoid shortcut solutions?](https://mlsanity.com/paper/2106.11230/) (2021)
- [ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding](https://mlsanity.com/paper/2109.04380/) (2021)
- [Prompt-Learning for Fine-Grained Entity Typing](https://mlsanity.com/paper/2108.10604/) (2021)

## Built on by (most cited first, 30 of 39)

- [Language models are not naysayers: An analysis of language models on negation benchmarks](https://mlsanity.com/paper/2306.08189/) (2023)
- [Contrastive Learning of Sentence Embeddings from Scratch](https://mlsanity.com/paper/2305.15077/) (2023)
- [Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervision](https://mlsanity.com/paper/2206.14719/) (2022)
- [Alleviating Over-smoothing for Unsupervised Sentence Representation](https://mlsanity.com/paper/2305.06154/) (2023)
- [Ranking-Enhanced Unsupervised Sentence Representation Learning](https://mlsanity.com/paper/2209.04333/) (2022)
- [Does Negative Sampling Matter? A Review with Insights into its Theory and Applications](https://mlsanity.com/paper/2402.17238/) (2024)
- [Composition-contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2307.07380/) (2023)
- [KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning](https://mlsanity.com/paper/2403.17486/) (2024)
- [MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languages](https://mlsanity.com/paper/2207.00758/) (2022)
- [SDA: Simple Discrete Augmentation for Contrastive Sentence Representation Learning](https://mlsanity.com/paper/2210.03963/) (2022)
- [CoT-BERT: Enhancing Unsupervised Sentence Representation through Chain-of-Thought](https://mlsanity.com/paper/2309.11143/) (2023)
- [Enhancing Semantics in Multimodal Chain of Thought via Soft Negative Sampling](https://mlsanity.com/paper/2405.09848/) (2024)
- [PaAno: Patch-Based Representation Learning for Time-Series Anomaly Detection](https://mlsanity.com/paper/2602.01359/) (2026)
- [Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding](https://mlsanity.com/paper/2211.03348/) (2022)
- [Clustering-Aware Negative Sampling for Unsupervised Sentence Representation](https://mlsanity.com/paper/2305.09892/) (2023)
- [Revealing the Blind Spot of Sentence Encoder Evaluation by HEROS](https://mlsanity.com/paper/2306.05083/) (2023)
- [DebCSE: Rethinking Unsupervised Contrastive Sentence Embedding Learning in the Debiasing Perspective](https://mlsanity.com/paper/2309.07396/) (2023)
- [DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning](https://mlsanity.com/paper/2401.13621/) (2024)
- [Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning](https://mlsanity.com/paper/2409.12887/) (2024)
- [Towards Ontology-Enhanced Representation Learning for Large Language Models](https://mlsanity.com/paper/2405.20527/) (2024)
- [CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Pass](https://mlsanity.com/paper/2505.00389/) (2025)
- [Block-SCL: Blocking Matters for Supervised Contrastive Learning in Product Matching](https://mlsanity.com/paper/2207.02008/) (2022)
- [Identical and Fraternal Twins: Fine-Grained Semantic Contrastive Learning of Sentence Representations](https://mlsanity.com/paper/2307.10932/) (2023)
- [HU at SemEval-2024 Task 8A: Can Contrastive Learning Learn Embeddings to Detect Machine-Generated Text?](https://mlsanity.com/paper/2402.11815/) (2024)
- [ZusammenQA: Data Augmentation with Specialized Models for Cross-lingual Open-retrieval Question Answering System](https://mlsanity.com/paper/2205.14981/) (2022)
- [ESCL: Equivariant Self-Contrastive Learning for Sentence Representations](https://mlsanity.com/paper/2303.05143/) (2023)
- [Adapting Sentence Transformers for the Aviation Domain](https://mlsanity.com/paper/2305.09556/) (2023)
- [HiCL: Hierarchical Contrastive Learning of Unsupervised Sentence Embeddings](https://mlsanity.com/paper/2310.09720/) (2023)
- [Domain Adaptation for Japanese Sentence Embeddings with Contrastive Learning based on Synthetic Sentence Generation](https://mlsanity.com/paper/2503.09094/) (2025)
- [D2CSE: Difference-aware Deep continuous prompts for Contrastive Sentence Embeddings](https://mlsanity.com/paper/2304.08991/) (2023)

## Abstract (arXiv)

Unsupervised sentence embedding aims to obtain the most appropriate embedding for a sentence to reflect its semantic. Contrastive learning has been attracting developing attention. For a sentence, current models utilize diverse data augmentation methods to generate positive samples, while consider other independent sentences as negative samples. Then they adopt InfoNCE loss to pull the embeddings of positive pairs gathered, and push those of negative pairs scattered. Although these models have made great progress on sentence embedding, we argue that they may suffer from feature suppression. The models fail to distinguish and decouple textual similarity and semantic similarity. And they may overestimate the semantic similarity of any pairs with similar textual regardless of the actual semantic difference between them. This is because positive pairs in unsupervised contrastive learning come with similar and even the same textual through data augmentation. To alleviate feature suppression, we propose contrastive learning for unsupervised sentence embedding with soft negative samples (SNCSE). Soft negative samples share highly similar textual but have surely and apparently different semantic with the original samples. Specifically, we take the negation of original sentences as soft negative samples, and propose Bidirectional Margin Loss (BML) to introduce them into traditional contrastive learning framework, which merely involves positive and negative samples. Our experimental results show that SNCSE can obtain state-of-the-art performance on semantic textual similarity (STS) task with average Spearman's correlation coefficient of 78.97% on BERTbase and 79.23% on RoBERTabase. Besides, we adopt rank-based error analysis method to detect the weakness of SNCSE for future study.