# Improving Contrastive Learning of Sentence Embeddings from AI Feedback

arXiv: https://arxiv.org/abs/2305.01918 · HTML: https://mlsanity.com/paper/2305.01918/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 32 of its 57 references that are in the map.
- First posted on arXiv 3 May 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2107.02137/) (2021)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks](https://mlsanity.com/paper/2010.08240/) (2020)
- [DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2204.10298/) (2022)
- [PromptBERT: Improving BERT Sentence Embeddings with Prompts](https://mlsanity.com/paper/2201.04337/) (2022)
- [ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding](https://mlsanity.com/paper/2109.04380/) (2021)
- [Debiased Contrastive Learning of Unsupervised Sentence Representations](https://mlsanity.com/paper/2205.00656/) (2022)
- [ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval](https://mlsanity.com/paper/2205.09153/) (2022)

## Built on by (most cited first, 29 of 29)

- [Unified Demonstration Retriever for In-Context Learning](https://mlsanity.com/paper/2305.04320/) (2023)
- [Scaling Sentence Embeddings with Large Language Models](https://mlsanity.com/paper/2307.16645/) (2023)
- [ClusterLLM: Large Language Models as a Guide for Text Clustering](https://mlsanity.com/paper/2305.14871/) (2023)
- [GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning](https://mlsanity.com/paper/2402.16829/) (2024)
- [Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost](https://mlsanity.com/paper/2306.15766/) (2023)
- [Evaluating Hallucinations in Chinese Large Language Models](https://mlsanity.com/paper/2310.03368/) (2023)
- [HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluation](https://mlsanity.com/paper/2406.07070/) (2024)
- [LLMvsSmall Model? Large Language Model Based Text Augmentation Enhanced Personality Detection Model](https://mlsanity.com/paper/2403.07581/) (2024)
- [Meta-Task Prompting Elicits Embeddings from Large Language Models](https://mlsanity.com/paper/2402.18458/) (2024)
- [Generalized Category Discovery with Large Language Models in the Loop](https://mlsanity.com/paper/2312.10897/) (2023)
- [Diffusion Models Through a Global Lens: Are They Culturally Inclusive?](https://mlsanity.com/paper/2502.08914/) (2025)
- [Language Models are Universal Embedders](https://mlsanity.com/paper/2310.08232/) (2023)
- [Controllable Data Augmentation for Few-Shot Text Mining with Chain-of-Thought Attribute Manipulation](https://mlsanity.com/paper/2307.07099/) (2023)
- [GLEAN: Active Generalized Category Discovery with Diverse LLM Feedback](https://mlsanity.com/paper/2502.18414/) (2025)
- [CSRv2: Unlocking Ultra-Sparse Embeddings](https://mlsanity.com/paper/2602.05735/) (2026)
- [ZYN: Zero-Shot Reward Models with Yes-No Questions for RLAIF](https://mlsanity.com/paper/2308.06385/) (2023)
- [DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning](https://mlsanity.com/paper/2401.13621/) (2024)
- [Large Language Model Augmented Exercise Retrieval for Personalized Language Learning](https://mlsanity.com/paper/2402.16877/) (2024)
- [Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning](https://mlsanity.com/paper/2409.12887/) (2024)
- [Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning](https://mlsanity.com/paper/2310.10962/) (2023)
- [Universal New Physics Latent Space](https://mlsanity.com/paper/2407.20315/) (2024)
- [On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey](https://mlsanity.com/paper/2507.20783/) (2025)
- [Exploring ChatGPT-based Augmentation Strategies for Contrastive Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2409.11218/) (2024)
- [DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learning](https://mlsanity.com/paper/2506.21096/) (2025)
- [BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation](https://mlsanity.com/paper/2508.06781/) (2025)
- [Template-assisted Contrastive Learning of Task-oriented Dialogue Sentence Embeddings](https://mlsanity.com/paper/2305.14299/) (2023)
- [Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model](https://mlsanity.com/paper/2309.06453/) (2023)
- [Navigation-GPT: A Robust and Adaptive Framework Utilizing Large Language Models for Navigation Applications](https://mlsanity.com/paper/2502.16402/) (2025)
- [ttda704 at SemEval-2026 Task 4: Modeling Narrative Structures via Pseudonymization and Multi-View Sentence Alignment](https://mlsanity.com/paper/2606.15783/) (2026)

## Abstract (arXiv)

Contrastive learning has become a popular approach in natural language processing, particularly for the learning of sentence embeddings. However, the discrete nature of natural language makes it difficult to ensure the quality of positive and negative sample pairs generated through data augmentation methods. Although supervised contrastive learning can produce more accurate sample pairs with human feedback labels, it still lacks fine-grained training signals. In this paper, we propose to improve Contrastive Learning of sentence embeddings from AI Feedback (CLAIF). Our method utilizes AI feedback from large pre-trained language models (LLMs) to construct sample pairs with fine-grained sample similarity scores to improve contrastive learning. Besides, we combine human feedback and AI feedback to provide better supervision signals for supervised contrastive learning of sentence embeddings. Experimental results show that our method achieves state-of-the-art performance on several semantic textual similarity (STS) and transfer learning tasks compared to other unsupervised and supervised contrastive learning methods.