# Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning

arXiv: https://arxiv.org/abs/2310.10962 · HTML: https://mlsanity.com/paper/2310.10962/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 54 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [In-context Examples Selection for Machine Translation](https://mlsanity.com/paper/2212.02437/) (2022)
- [DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2204.10298/) (2022)
- [Debiased Contrastive Learning of Unsupervised Sentence Representations](https://mlsanity.com/paper/2205.00656/) (2022)
- [Improving Contrastive Learning of Sentence Embeddings from AI Feedback](https://mlsanity.com/paper/2305.01918/) (2023)
- [InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2210.06432/) (2022)
- [Contrastive Learning of Sentence Embeddings from Scratch](https://mlsanity.com/paper/2305.15077/) (2023)
- [Measuring the Similarity of Sentential Arguments in Dialog](https://mlsanity.com/paper/1709.01887/) (2017)
- [RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank](https://mlsanity.com/paper/2305.16726/) (2023)

## Built on by (most cited first, 3 of 3)

- [Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning](https://mlsanity.com/paper/2409.12887/) (2024)
- [Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models](https://mlsanity.com/paper/2502.13656/) (2025)
- [Understanding InfoNCE: Transition Probability Matrix Induced Feature Clustering](https://mlsanity.com/paper/2511.12180/) (2025)

## Abstract (arXiv)

Recently, large language models (LLMs) have emerged as a groundbreaking technology and their unparalleled text generation capabilities have sparked interest in their application to the fundamental sentence representation learning task. Existing methods have explored utilizing LLMs as data annotators to generate synthesized data for training contrastive learning based sentence embedding models such as SimCSE. However, since contrastive learning models are sensitive to the quality of sentence pairs, the effectiveness of these methods is largely influenced by the content generated from LLMs, highlighting the need for more refined generation in the context of sentence representation learning. Building upon this premise, we propose MultiCSR, a multi-level contrastive sentence representation learning framework that decomposes the process of prompting LLMs to generate a corpus for training base sentence embedding models into three stages (i.e., sentence generation, sentence pair construction, in-batch training) and refines the generated content at these three distinct stages, ensuring only high-quality sentence pairs are utilized to train a base contrastive learning model. Our extensive experiments reveal that MultiCSR enables a less advanced LLM to surpass the performance of ChatGPT, while applying it to ChatGPT achieves better state-of-the-art results. Comprehensive analyses further underscore the potential of our framework in various application scenarios and achieving better sentence representation learning with LLMs.