mlsanityData to 7 Oct 2026

M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation

arXiv 2402.03216 ↗First posted 5 Feb 2024cs.CL cs.AI cs.LGcited by 1,050

Not on the map. Why?

Cited by 245 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we introduce a new embedding model called M3-Embedding, which is distinguished for its versatility in textitMulti-Linguality, textitMulti-Functionality, and textitMulti-Granularity. It provides a uniform support for the semantic retrieval of more than 100 working languages. It can simultaneously accomplish the three common retrieval functionalities: dense retrieval, multi-vector retrieval, and sparse retrieval. Besides, it is also capable of processing inputs of different granularities, spanning from short sentences to long documents of up to 8,192 tokens. The effective training of M3-Embedding presents a series of technical contributions. Notably, we propose a novel self-knowledge distillation approach, where the relevance scores from different retrieval functionalities can be integrated as the teacher signal to enhance the training quality. We also optimize the batching strategy, which enables a large batch size and high training throughput to improve the discriminativeness of embeddings. M3-Embedding exhibits a superior performance in our experiment, leading to new state-of-the-art results on multilingual, cross-lingual, and long-document retrieval benchmarks.

Key facts. Cited by 1,050 papers in the map (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

43 of 57 references.

References · 43

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
31 more
  1. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  2. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  4. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  5. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  6. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  8. 2019How multilingual is Multilingual BERT?cited by 642
  9. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  10. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  11. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  12. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  13. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  14. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  15. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  16. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  17. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  18. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  19. 2019Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoringcited by 164
  20. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
  21. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  22. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  23. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  24. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  25. 2017DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applicationscited by 110
  26. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  27. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  28. 2021A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniquescited by 69
  29. 2023T2Ranking: A large-scale Chinese Benchmark for Passage Rankingcited by 59
  30. 2023LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Datasetcited by 37
  31. 2022Towards Best Practices for Training Multilingual Dense Retrieval Modelscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 134; 2025: 414; 2026: 502.

1,050 papers.

Sort:

By area:

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  2. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  3. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  4. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  5. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  6. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  7. 2024Generative Representational Instruction Tuningcited by 227
  8. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  9. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  10. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 163
  11. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  12. 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecyclecited by 129
38 more
  1. 2025Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memorycited by 129
  2. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  3. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 118
  4. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 104
  5. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 85
  6. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  7. 2024Making Text Embedders Few-Shot Learnerscited by 78
  8. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 67
  9. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
  10. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  11. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selectioncited by 64
  12. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 60
  13. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 59
  14. 2024Arctic-Embed 2.0: Multilingual Retrieval Without Compromisecited by 57
  15. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 56
  16. 2025LongCat-Flash Technical Reportcited by 54
  17. 2026SkillNet: Create, Evaluate, and Connect AI Skillscited by 48
  18. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  19. 2025LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?cited by 47
  20. 2024Jina CLIP: Your CLIP Model Is Also Your Text Retrievercited by 46
  21. 2024RS-Agent: Automating Remote Sensing Tasks through Intelligent Agentcited by 46
  22. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 46
  23. 2024Is Semantic Chunking Worth the Computational Cost?cited by 43
  24. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  25. 2025General Agentic Memory Via Deep Researchcited by 42
  26. 2024Memory3: Language Modeling with Explicit Memorycited by 39
  27. 2024RAGEval: Scenario Specific RAG Evaluation Dataset Generation Frameworkcited by 39
  28. 2025Agentic Entropy-Balanced Policy Optimizationcited by 39
  29. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  30. 2025TrustRAG: Enhancing Robustness and Trustworthiness in Retrieval-Augmented Generationcited by 37
  31. 2024jina-clip-v2: Multilingual Multimodal Embeddings for Text and Imagescited by 36
  32. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 35
  33. 2025REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmarkcited by 34
  34. 2024RAR-b: Reasoning as Retrieval Benchmarkcited by 33
  35. 2024CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrievalcited by 33
  36. 2025PersonaX: A Recommendation Agent Oriented User Modeling Framework for Long Behavior Sequencecited by 33
  37. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  38. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31

50 most cited of 1,050 All 1,050 (JSON)