mlsanityData to 7 Oct 2026

Improving Text Embeddings with Large Language Models

arXiv 2401.00368 ↗First posted 31 Dec 2023cs.CL cs.IRcited by 295

Not on the map. Why?

Cited by 40 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. Unlike existing methods that often depend on multi-stage intermediate pre-training with billions of weakly-supervised text pairs, followed by fine-tuning with a few labeled datasets, our method does not require building complex training pipelines or relying on manually collected datasets that are often constrained by task diversity and language coverage. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across 93 languages. We then fine-tune open-source decoder-only LLMs on the synthetic data using standard contrastive loss. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data. Furthermore, when fine-tuned with a mixture of synthetic and labeled data, our model sets new state-of-the-art results on the BEIR and MTEB benchmarks.

Key facts. Cited by 295 papers in the map (data through 7 Oct 2026). Builds on 46 of its 72 references that are in the map. First posted on arXiv 31 Dec 2023.

Builds on

46 of 72 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  12. 2023Mistral 7Bcited by 3,065
34 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  4. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  5. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  6. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  7. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  9. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  10. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  11. 2019ELI5: Long Form Question Answeringcited by 567
  12. 2023Textbooks Are All You Needcited by 561
  13. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  14. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  15. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  16. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  17. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  18. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  19. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  20. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  21. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  22. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  23. 2019Document Expansion by Query Predictioncited by 225
  24. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  25. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  26. 2021Generating Datasets with Pretrained Language Modelscited by 192
  27. 2023Query2doc: Query Expansion with Large Language Modelscited by 146
  28. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  29. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  30. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  31. 2023AnglE-optimized Text Embeddingscited by 98
  32. 2023T2Ranking: A large-scale Chinese Benchmark for Passage Rankingcited by 59
  33. 2022DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Enginecited by 15
  34. 2023Language Models are Universal Embedderscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 94; 2025: 129; 2026: 72.

295 papers.

Sort:

By area:

  1. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  2. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  3. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  4. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  5. 2024Generative Representational Instruction Tuningcited by 227
  6. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  7. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  8. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  9. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  10. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
  11. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  12. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
38 more
  1. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  2. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  3. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 128
  4. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 123
  5. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  6. 2025Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Spacecited by 109
  7. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 104
  8. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 98
  9. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 93
  10. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  11. 2024Towards Open-ended Visual Quality Comparisoncited by 85
  12. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  13. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
  14. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  15. 2024Making Text Embedders Few-Shot Learnerscited by 78
  16. 2024From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMscited by 77
  17. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 76
  18. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 72
  19. 2024Repetition Improves Language Model Embeddingscited by 70
  20. 2024Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Constructioncited by 70
  21. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 67
  22. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  23. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 67
  24. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 59
  25. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 59
  26. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
  27. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 52
  28. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  29. 2024CultureLLM: Incorporating Cultural Differences into Large Language Modelscited by 49
  30. 2025Chain-of-Retrieval Augmented Generationcited by 48
  31. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  32. 2024LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendationcited by 47
  33. 2025Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Taskscited by 46
  34. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 44
  35. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  36. 2024Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Modelscited by 39
  37. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  38. 2024CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Rerankingcited by 37

50 most cited of 295 All 295 (JSON)