# Let LLMs Speak Embedding Languages: Generative Text Embeddings via Iterative Contrastive Refinement

arXiv: https://arxiv.org/abs/2509.24291 · HTML: https://mlsanity.com/paper/2509.24291/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 24 of 24)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Improving Text Embeddings with Large Language Models](https://mlsanity.com/paper/2401.00368/) (2023)
- [Generative Representational Instruction Tuning](https://mlsanity.com/paper/2402.09906/) (2024)
- [MMTEB: Massive Multilingual Text Embedding Benchmark](https://mlsanity.com/paper/2502.13595/) (2025)
- [A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression](https://mlsanity.com/paper/2406.11430/) (2024)
- [Making Text Embedders Few-Shot Learners](https://mlsanity.com/paper/2409.15700/) (2024)
- [FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions](https://mlsanity.com/paper/2403.15246/) (2024)
- [ONCE: Boosting Content-based Recommendation with Both Open- and Closed-source Large Language Models](https://mlsanity.com/paper/2305.06566/) (2023)
- [LLMs are Also Effective Embedding Models: An In-depth Overview](https://mlsanity.com/paper/2412.12591/) (2024)
- [Answer is All You Need: Instruction-following Text Embedding via Answering the Question](https://mlsanity.com/paper/2402.09642/) (2024)

## Built on by (most cited first, 10 of 10)

- [LaSER: Internalizing Explicit Reasoning into Latent Space for Dense Retrieval](https://mlsanity.com/paper/2603.01425/) (2026)
- [Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval](https://mlsanity.com/paper/2602.19961/) (2026)
- [CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding](https://mlsanity.com/paper/2601.21262/) (2026)
- [Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddings](https://mlsanity.com/paper/2601.11124/) (2026)
- [LLM2Vec-Gen: Generative Embeddings from Large Language Models](https://mlsanity.com/paper/2603.10913/) (2026)
- [PromptEmbedder: Efficient and Transferable Text Embedding via Dual-LLM Soft Prompting](https://mlsanity.com/paper/2605.28066/) (2026)
- [SHIFT: Self-reconstruction Harnesses Implicit Fine-grained Thinking for Retrieval](https://mlsanity.com/paper/2607.21333/) (2026)
- [Retrieval Grounding Latent Reasoning for Dense Retrieval](https://mlsanity.com/paper/2608.14107/) (2026)
- [Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddings](https://mlsanity.com/paper/2609.15296/) (2026)
- [A Survey on Fake Review Detection: From Pre-trained Language Models to Large Language Models](https://mlsanity.com/paper/2609.30292/) (2026)

## Abstract (arXiv)

Existing large language model (LLM)-based embeddings typically adopt an encoder-only paradigm, treating LLMs as static feature extractors and overlooking their core generative strengths. We introduce GIRCSE (Generative Iterative Refinement for Contrastive Sentence Embeddings), a novel framework that leverages autoregressive generation to iteratively refine semantic representations. By producing sequences of soft tokens optimized under contrastive objective, GIRCSE captures latent concepts and implicit semantics that encoder-only methods often miss. To guide this process, we propose an Iterative Contrastive Refinement (ICR) objective that encourages each refinement step to yield better representations. Extensive experiments show that GIRCSE outperforms strong LLM-based embedding baselines on the MTEB benchmark and instruction-following tasks. Moreover, GIRCSE exhibits an emergent test-time scaling property: generating more tokens at inference steadily improves embedding quality. Our results establish generative iterative refinement as a new paradigm for representation learning.