mlsanityData to 7 Oct 2026

Text and Code Embeddings by Contrastive Pre-Training

arXiv 2201.10005 ↗First posted 24 Jan 2022cs.CL cs.LGcited by 358

Not on the map. Why?

Cited by 31 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Text embeddings are useful features in many applications such as semantic search and computing text similarity. Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture. In this work, we show that contrastive pre-training on unsupervised data at scale leads to high quality vector representations of text and code. The same unsupervised text embeddings that achieve new state-of-the-art results in linear-probe classification also display impressive semantic search capabilities and sometimes even perform competitively with fine-tuned models. On linear-probe classification accuracy averaging over 7 tasks, our best unsupervised model achieves a relative improvement of 4% and 1.8% over previous best unsupervised and supervised text embedding models respectively. The same text embeddings when evaluated on large-scale semantic search attains a relative improvement of 23.4%, 14.7%, and 10.6% over previous best unsupervised methods on MSMARCO, Natural Questions and TriviaQA benchmarks, respectively. Similarly to text embeddings, we train code embedding models on (text, code) pairs, obtaining a 20.8% relative improvement over prior best work on code search.

Key facts. Cited by 358 papers in the map (data through 7 Oct 2026). Builds on 64 of its 87 references that are in the map. First posted on arXiv 24 Jan 2022.

Builds on

64 of 87 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  6. 2020Supervised Contrastive Learningcited by 2,803
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  13. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  14. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  15. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  16. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  17. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  18. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  19. 2019Contrastive Multiview Codingcited by 1,297
  20. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  21. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  22. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  23. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  24. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  25. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  26. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  27. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  28. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  29. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  30. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  31. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  32. 2018Gender Bias in Coreference Resolutioncited by 438
  33. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  34. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  35. 2015Skip-Thought Vectorscited by 321
  36. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  37. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  38. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 26; 2023: 75; 2024: 142; 2025: 84; 2026: 31.

358 papers.

Sort:

By area:

  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  2. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  3. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  4. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  5. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  6. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  7. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  8. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  9. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  10. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  11. 2023Improving Text Embeddings with Large Language Modelscited by 295
  12. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
38 more
  1. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  2. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  3. 2024Generative Representational Instruction Tuningcited by 227
  4. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  5. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 189
  6. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
  7. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  8. 2023Representation Learning with Large Language Models for Recommendationcited by 155
  9. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  10. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  11. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  12. 2024Homogenization Effects of Large Language Models on Human Creative Ideationcited by 127
  13. 2023Training Language Models with Language Feedback at Scalecited by 125
  14. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
  15. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  16. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  17. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
  18. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 93
  19. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 92
  20. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
  21. 2023Benchmarking large language models for biomedical natural language processing applications and recommendationscited by 78
  22. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 77
  23. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  24. 2023RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Modelscited by 69
  25. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
  26. 2023A Multi-Task Embedder For Retrieval Augmented LLMscited by 63
  27. 2024HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modelingcited by 63
  28. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 60
  29. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 59
  30. 2023ChatHaruhi: Reviving Anime Character in Reality via Large Language Modelcited by 57
  31. 2023Grounding Complex Natural Language Commands for Temporal Tasks in Unseen Environmentscited by 52
  32. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  33. 2022Training Language Models with Language Feedbackcited by 48
  34. 2023Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Modelscited by 48
  35. 2022Questions Are All You Need to Train a Dense Passage Retrievercited by 47
  36. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  37. 2023Dataset Quantizationcited by 47
  38. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47

50 most cited of 358 All 358 (JSON)