mlsanityData to 7 Oct 2026

Contrastive Learning of Sentence Embeddings from Scratch

arXiv 2305.15077 ↗First posted 24 May 2023cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Contrastive learning has been the dominant approach to train state-of-the-art sentence embeddings. Previous studies have typically learned sentence embeddings either through the use of human-annotated natural language inference (NLI) data or via large-scale unlabeled sentences in an unsupervised manner. However, even in the case of unlabeled data, their acquisition presents challenges in certain domains due to various reasons. To address these issues, we present SynCSE, a contrastive learning framework that trains sentence embeddings with synthesized data. Specifically, we explore utilizing large language models to synthesize the required data samples for contrastive learning, including (1) producing positive and negative annotations given unlabeled sentences (SynCSE-partial), and (2) generating sentences along with their corresponding annotations from scratch (SynCSE-scratch). Experimental results on sentence similarity and reranking tasks indicate that both SynCSE-partial and SynCSE-scratch greatly outperform unsupervised baselines, and SynCSE-partial even achieves comparable performance to the supervised models in most settings.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 30 of its 59 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

30 of 59 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  10. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  11. 2018Universal Sentence Encodercited by 485
  12. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
18 more
  1. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  2. 2021Generating Datasets with Pretrained Language Modelscited by 192
  3. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  4. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 73
  5. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  6. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  7. 2022SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samplescited by 39
  8. 2022Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learningcited by 30
  9. 2022PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddingscited by 29
  10. 2015Semi-supervised Question Retrieval with Gated Convolutionscited by 28
  11. 2017Measuring the Similarity of Sentential Arguments in Dialogcited by 23
  12. 2022Just Rank: Rethinking Evaluation with Word and Sentence Similaritiescited by 20
  13. 2022Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Frameworkcited by 18
  14. 2022Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negativescited by 16
  15. 2022Sentence Representation Learning with Generative Objective rather than Contrastive Objectivecited by 12
  16. 2022Differentiable Data Augmentation for Contrastive Sentence Representation Learningcited by 9
  17. 2022Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embeddingcited by 6
  18. 2023Instance Smoothed Contrastive Learning for Unsupervised Sentence Embeddingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 13; 2025: 8; 2026: 1.

24 papers.

Sort:

By area:

  1. 2023Scaling Sentence Embeddings with Large Language Modelscited by 83
  2. 2024Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QAcited by 28
  3. 2024Meta-Task Prompting Elicits Embeddings from Large Language Modelscited by 15
  4. 2024GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddingscited by 14
  5. 2024Little Giants: Synthesizing High-Quality Embedding Data at Scalecited by 11
  6. 2024Unleashing the Power of Large Language Models in Zero-shot Relation Extraction via Self-Promptingcited by 10
  7. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  8. 2024Subgraph-Aware Training of Language Models for Knowledge Graph Completion Using Structure-Aware Contrastive Learningcited by 4
  9. 2024Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learningcited by 4
  10. 2023Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learningcited by 3
  11. 2024TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Modelscited by 3
  12. 2024Towards Better Code Understanding in Decoder-Only Models with Contrastive Learningcited by 3
12 more
  1. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  2. 2024Ruri: Japanese General Text Embeddingscited by 2
  3. 2025Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Modelscited by 2
  4. 2024Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Modelscited by 1
  5. 2025Domain Adaptation for Japanese Sentence Embeddings with Contrastive Learning based on Synthetic Sentence Generationcited by 1
  6. 2025BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillationcited by 1
  7. 2024DefSent+: Improving sentence embeddings of language models by projecting definition sentences into a quasi-isotropic or isotropic vector space of unlimited dictionary entriescited by 0
  8. 2024Are BabyLMs Second Language Learners?cited by 0
  9. 2025FinCPRG: A Bidirectional Generation Pipeline for Hierarchical Queries and Rich Relevance in Financial Chinese Passage Retrievalcited by 0
  10. 2025Cropping outperforms dropout as an augmentation strategy for self-supervised training of text embeddingscited by 0
  11. 2025Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NERcited by 0
  12. 2026Bootstrapping Embeddings for Low Resource Languagescited by 0