mlsanityData to 7 Oct 2026

Text Embeddings by Weakly-Supervised Contrastive Pre-training

arXiv 2212.03533 ↗First posted 7 Dec 2022cs.CL cs.IRcited by 1,216

Not on the map. Why?

Cited by 248 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision signals from our curated large-scale text pair dataset (called CCPairs). E5 can be readily used as a general-purpose embedding model for any tasks requiring a single-vector representation of texts such as retrieval, clustering, and classification, achieving strong performance in both zero-shot and fine-tuned settings. We conduct extensive evaluations on 56 datasets from the BEIR and MTEB benchmarks. For zero-shot settings, E5 is the first model that outperforms the strong BM25 baseline on the BEIR retrieval benchmark without using any labeled data. When fine-tuned, E5 obtains the best results on the MTEB benchmark, beating existing embedding models with 40x more parameters.

Key facts. Cited by 1,216 papers in the map (data through 7 Oct 2026). Builds on 49 of its 66 references that are in the map. First posted on arXiv 7 Dec 2022.

Builds on

49 of 66 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
37 more
  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  4. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  6. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  7. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  8. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  9. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  10. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  11. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  12. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  13. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  14. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  15. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  16. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  17. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  18. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  19. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  20. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  21. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  22. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  23. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  24. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  25. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  26. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  27. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  28. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 195
  29. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  30. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  31. 2019SELF: Learning to Filter Noisy Labels with Self-Ensemblingcited by 164
  32. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  33. 2020Learning Dense Representations of Phrases at Scalecited by 91
  34. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  35. 2021Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?cited by 59
  36. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  37. 2022LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrievalcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 70; 2024: 261; 2025: 411; 2026: 471.

1,216 papers.

Sort:

By area:

  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  2. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  3. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  4. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  5. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  6. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  7. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  8. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  9. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  10. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  11. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  12. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
38 more
  1. 2023Improving Text Embeddings with Large Language Modelscited by 295
  2. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  3. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  4. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
  5. 2023One for All: Towards Training One Graph Model for All Classification Taskscited by 239
  6. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  7. 2024Generative Representational Instruction Tuningcited by 227
  8. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  9. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  10. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  11. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  12. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  13. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  14. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
  15. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168
  16. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  17. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 165
  18. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  19. 2023Query2doc: Query Expansion with Large Language Modelscited by 146
  20. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  21. 2023Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learningcited by 132
  22. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 128
  23. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 123
  24. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  25. 2023Retrieval meets Long Context Large Language Modelscited by 114
  26. 2024Knowledge Fusion of Large Language Modelscited by 110
  27. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  28. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 102
  29. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 98
  30. 2024Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answeringcited by 95
  31. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 94
  32. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 93
  33. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  34. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
  35. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  36. 2024ZeroG: Investigating Cross-dataset Zero-shot Transferability in Graphscited by 83
  37. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  38. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 83

50 most cited of 1,216 All 1,216 (JSON)