# Contrastive Learning of Sentence Embeddings from Scratch

arXiv: https://arxiv.org/abs/2305.15077 · HTML: https://mlsanity.com/paper/2305.15077/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 59 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [PromptBERT: Improving BERT Sentence Embeddings with Prompts](https://mlsanity.com/paper/2201.04337/) (2022)
- [ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding](https://mlsanity.com/paper/2109.04380/) (2021)
- [ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback](https://mlsanity.com/paper/2210.12329/) (2022)
- [Debiased Contrastive Learning of Unsupervised Sentence Representations](https://mlsanity.com/paper/2205.00656/) (2022)
- [SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples](https://mlsanity.com/paper/2201.05979/) (2022)
- [Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning](https://mlsanity.com/paper/2203.06875/) (2022)
- [PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings](https://mlsanity.com/paper/2201.12093/) (2022)
- [Semi-supervised Question Retrieval with Gated Convolutions](https://mlsanity.com/paper/1512.05726/) (2015)
- [Measuring the Similarity of Sentential Arguments in Dialog](https://mlsanity.com/paper/1709.01887/) (2017)
- [Just Rank: Rethinking Evaluation with Word and Sentence Similarities](https://mlsanity.com/paper/2203.02679/) (2022)
- [Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework](https://mlsanity.com/paper/2210.16798/) (2022)
- [Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives](https://mlsanity.com/paper/2206.02457/) (2022)
- [Sentence Representation Learning with Generative Objective rather than Contrastive Objective](https://mlsanity.com/paper/2210.08474/) (2022)
- [Differentiable Data Augmentation for Contrastive Sentence Representation Learning](https://mlsanity.com/paper/2210.16536/) (2022)
- [Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding](https://mlsanity.com/paper/2211.03348/) (2022)
- [Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding](https://mlsanity.com/paper/2305.07424/) (2023)

## Built on by (most cited first, 24 of 24)

- [Scaling Sentence Embeddings with Large Language Models](https://mlsanity.com/paper/2307.16645/) (2023)
- [Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QA](https://mlsanity.com/paper/2407.15353/) (2024)
- [Meta-Task Prompting Elicits Embeddings from Large Language Models](https://mlsanity.com/paper/2402.18458/) (2024)
- [GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings](https://mlsanity.com/paper/2410.14635/) (2024)
- [Little Giants: Synthesizing High-Quality Embedding Data at Scale](https://mlsanity.com/paper/2410.18634/) (2024)
- [Unleashing the Power of Large Language Models in Zero-shot Relation Extraction via Self-Prompting](https://mlsanity.com/paper/2410.01154/) (2024)
- [Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning](https://mlsanity.com/paper/2506.08354/) (2025)
- [Subgraph-Aware Training of Language Models for Knowledge Graph Completion Using Structure-Aware Contrastive Learning](https://mlsanity.com/paper/2407.12703/) (2024)
- [Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning](https://mlsanity.com/paper/2409.12887/) (2024)
- [Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning](https://mlsanity.com/paper/2310.10962/) (2023)
- [TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2401.06620/) (2024)
- [Towards Better Code Understanding in Decoder-Only Models with Contrastive Learning](https://mlsanity.com/paper/2406.12326/) (2024)
- [On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey](https://mlsanity.com/paper/2507.20783/) (2025)
- [Ruri: Japanese General Text Embeddings](https://mlsanity.com/paper/2409.07737/) (2024)
- [Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models](https://mlsanity.com/paper/2502.13656/) (2025)
- [BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation](https://mlsanity.com/paper/2508.06781/) (2025)
- [Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models](https://mlsanity.com/paper/2401.08491/) (2024)
- [Domain Adaptation for Japanese Sentence Embeddings with Contrastive Learning based on Synthetic Sentence Generation](https://mlsanity.com/paper/2503.09094/) (2025)
- [DefSent+: Improving sentence embeddings of language models by projecting definition sentences into a quasi-isotropic or isotropic vector space of unlimited dictionary entries](https://mlsanity.com/paper/2405.16153/) (2024)
- [Are BabyLMs Second Language Learners?](https://mlsanity.com/paper/2410.21254/) (2024)
- [FinCPRG: A Bidirectional Generation Pipeline for Hierarchical Queries and Rich Relevance in Financial Chinese Passage Retrieval](https://mlsanity.com/paper/2508.02222/) (2025)
- [Cropping outperforms dropout as an augmentation strategy for self-supervised training of text embeddings](https://mlsanity.com/paper/2508.03453/) (2025)
- [Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER](https://mlsanity.com/paper/2510.07566/) (2025)
- [Bootstrapping Embeddings for Low Resource Languages](https://mlsanity.com/paper/2603.01732/) (2026)

## Abstract (arXiv)

Contrastive learning has been the dominant approach to train state-of-the-art sentence embeddings. Previous studies have typically learned sentence embeddings either through the use of human-annotated natural language inference (NLI) data or via large-scale unlabeled sentences in an unsupervised manner. However, even in the case of unlabeled data, their acquisition presents challenges in certain domains due to various reasons. To address these issues, we present SynCSE, a contrastive learning framework that trains sentence embeddings with synthesized data. Specifically, we explore utilizing large language models to synthesize the required data samples for contrastive learning, including (1) producing positive and negative annotations given unlabeled sentences (SynCSE-partial), and (2) generating sentences along with their corresponding annotations from scratch (SynCSE-scratch). Experimental results on sentence similarity and reranking tasks indicate that both SynCSE-partial and SynCSE-scratch greatly outperform unsupervised baselines, and SynCSE-partial even achieves comparable performance to the supervised models in most settings.