mlsanityData to 7 Oct 2026

Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark

arXiv 2406.01607 ↗First posted 27 May 2024cs.IR cs.AI cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Text embedding methods have become increasingly popular in both industrial and academic fields due to their critical role in a variety of natural language processing tasks. The significance of universal text embeddings has been further highlighted with the rise of Large Language Models (LLMs) applications such as Retrieval-Augmented Systems (RAGs). While previous models have attempted to be general-purpose, they often struggle to generalize across tasks and domains. However, recent advancements in training data quantity, quality and diversity; synthetic data generation from LLMs as well as using LLMs as backbones encourage great improvements in pursuing universal text embeddings. In this paper, we provide an overview of the recent advances in universal text embedding models with a focus on the top performing text embeddings on Massive Text Embedding Benchmark (MTEB). Through detailed comparison and analysis, we highlight the key contributions and limitations in this area, and propose potentially inspiring future research directions.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 85 of its 125 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

85 of 125 references.

References · 85

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  11. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  12. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023Mistral 7Bcited by 3,065
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  13. 2018Deep contextualized word representationscited by 2,285
  14. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  15. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  16. 2024Mixtral of Expertscited by 1,785
  17. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  18. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  19. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  20. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  21. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  22. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  23. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  24. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  25. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  26. 2019RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Spacecited by 893
  27. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  28. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  29. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  30. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  31. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  32. 2019ELI5: Long Form Question Answeringcited by 567
  33. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  34. 2018Universal Sentence Encodercited by 485
  35. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  36. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  37. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  38. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 3.

19 papers.

Sort:

By area:

  1. 2024Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworkscited by 41
  2. 2024Writing Style Matters: An Examination of Bias and Fairness in Information Retrieval Systemscited by 15
  3. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  4. 2025Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applicationscited by 7
  5. 2025Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarkscited by 7
  6. 2025Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generationcited by 5
  7. 2025VN-MTEB: Vietnamese Massive Text Embedding Benchmarkcited by 5
  8. 2025Preserving Task-Relevant Information Under Linear Concept Removalcited by 3
  9. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  10. 2024Using Large Multimodal Models to Extract Knowledge Components for Knowledge Tracing from Multimedia Question Informationcited by 2
  11. 2025Disentangling Linguistic Features with Dimension-Wise Analysis of Vector Embeddingscited by 2
  12. 2025A Large-Language-Model Framework for Automated Humanitarian Situation Reportingcited by 2
7 more
  1. 2026When LLMs Imagine People: A Human-Centered Persona Brainstorm Audit for Bias and Fairness in Creative Applicationscited by 2
  2. 2025Semantic Adapter for Universal Text Embeddings: Diagnosing and Mitigating Negation Blindness to Enhance Universalitycited by 1
  3. 2025Resource-Efficient Adaptation of Large Language Models for Text Embeddings via Prompt Engineering and Contrastive Fine-tuningcited by 1
  4. 2025Diverse And Private Synthetic Datasets Generation for RAG evaluation: A multi-agent frameworkcited by 1
  5. 2025Keeping it Local, Tiny and Real: Automated Report Generation on Edge Computing Devices for Mechatronic-Based Cognitive Systemscited by 1
  6. 2024[Vision Paper] PRObot: Enhancing Patient-Reported Outcome Measures for Diabetic Retinopathy using Chatbots and Generative AIcited by 0
  7. 2026Progressive Searching for Retrieval in RAGcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FaMTEB: Massive Text Embedding Benchmark in Persian Language
  2. 2025Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective
  3. 2025Dewey Long Context Embedding Model: A Technical Report
  4. 2025Hakim: Farsi Text Embedding Model
  5. 2026Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings
  6. 2025Understanding the Influence of Synthetic Data for Text Embedders
  7. 2025GEM: Empowering LLM for both Embedding Generation and Language Understanding
  8. 2024A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokens