mlsanityData to 7 Oct 2026

Towards General Text Embeddings with Multi-stage Contrastive Learning

arXiv 2308.03281 ↗First posted 7 Aug 2023cs.CLcited by 705

Not on the map. Why?

Cited by 134 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present GTE, a general-purpose text embedding model trained with multi-stage contrastive learning. In line with recent advancements in unifying various NLP tasks into a single format, we train a unified text embedding model by employing contrastive learning over a diverse mixture of datasets from multiple sources. By significantly increasing the number of training data during both unsupervised pre-training and supervised fine-tuning stages, we achieve substantial performance gains over existing embedding models. Notably, even with a relatively modest parameter count of 110M, GTEbase outperforms the black-box embedding API provided by OpenAI and even surpasses 10x larger text embedding models on the massive text embedding benchmark. Furthermore, without additional fine-tuning on each programming language individually, our model outperforms previous best code retrievers of similar size by treating code as text. In summary, our model achieves impressive results by effectively harnessing multi-stage contrastive learning, offering a powerful and efficient text embedding model with broad applicability across various NLP and code-related tasks.

Key facts. Cited by 705 papers in the map (data through 7 Oct 2026). Builds on 52 of its 66 references that are in the map. First posted on arXiv 7 Aug 2023.

Builds on

52 of 66 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
38 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  2. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  3. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  5. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  6. 2017Mixed Precision Trainingcited by 1,135
  7. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  8. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  9. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  10. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  11. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  13. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  14. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  15. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  16. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  17. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  18. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  19. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  20. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  21. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  22. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  23. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  24. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  25. 2019Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoringcited by 164
  26. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  27. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  28. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  29. 2022Task-aware Retrieval with Instructionscited by 107
  30. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  31. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  32. 2021PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrievalcited by 50
  33. 2021Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Taskscited by 32
  34. 2022Multi-CPR: A Multi Domain Chinese Dataset for Passage Retrievalcited by 26
  35. 2022CodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Searchcited by 21
  36. 2022Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answeringcited by 14
  37. 2023SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negativescited by 9
  38. 2022Retrieval Oriented Masking Pre-training Language Model for Dense Passage Retrievalcited by 4

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 175; 2025: 314; 2026: 198.

705 papers.

Sort:

By area:

  1. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  2. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  3. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  4. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  5. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  6. 2023Improving Text Embeddings with Large Language Modelscited by 295
  7. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  8. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
  9. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  10. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  11. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
  12. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
38 more
  1. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  2. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  3. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  4. 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agentscited by 102
  5. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  6. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  7. 2024NV-Retriever: Improving text embedding models with effective hard-negative miningcited by 83
  8. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
  9. 2024Making Text Embedders Few-Shot Learnerscited by 78
  10. 2024Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generationcited by 78
  11. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  12. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 72
  13. 2024Repetition Improves Language Model Embeddingscited by 70
  14. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  15. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 63
  16. 2023RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Modelscited by 62
  17. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 60
  18. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 60
  19. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 59
  20. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 57
  21. 2025TxAgent: An AI Agent for Therapeutic Reasoning Across a Universe of Toolscited by 57
  22. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
  23. 2026SkillRouter: Skill Routing for LLM Agents at Scalecited by 57
  24. 2024FinTextQA: A Dataset for Long-form Financial Question Answeringcited by 54
  25. 2025Harnessing the Universal Geometry of Embeddingscited by 54
  26. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 52
  27. 2024DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learningcited by 52
  28. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inferencecited by 51
  29. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  30. 2025Text-to-LoRA: Instant Transformer Adaptioncited by 50
  31. 2024Don't Forget to Connect! Improving RAG with Graph-based Rerankingcited by 49
  32. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 48
  33. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  34. 2025SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and Multi-dimensional Evaluation for Automated Survey Writingcited by 47
  35. 2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generationcited by 46
  36. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 46
  37. 2025Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Taskscited by 46
  38. 2025Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalizationcited by 44

50 most cited of 705 All 705 (JSON)