mlsanityData to 7 Oct 2026

mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval

arXiv 2407.19669 ↗First posted 29 Jul 2024cs.CL cs.IRcited by 233

Not on the map. Why?

Cited by 41 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present systematic efforts in building long-context multilingual text representation model (TRM) and reranker from scratch for text retrieval. We first introduce a text encoder (base size) enhanced with RoPE and unpadding, pre-trained in a native 8192-token context (longer than 512 of previous multilingual encoders). Then we construct a hybrid TRM and a cross-encoder reranker by contrastive learning. Evaluations show that our text encoder outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM and reranker match the performance of large-sized state-of-the-art BGE-M3 models and achieve better results on long-context retrieval benchmarks. Further analysis demonstrate that our proposed models exhibit higher efficiency during both training and inference. We believe their efficiency and effectiveness could benefit various researches and industrial applications.

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). Builds on 65 of its 90 references that are in the map. First posted on arXiv 29 Jul 2024.

Builds on

65 of 90 references.

References · 65

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2020GLU Variants Improve Transformercited by 1,724
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  7. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  10. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  13. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  14. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  15. 2018Neural Network Acceptability Judgmentscited by 952
  16. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  17. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  18. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  19. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  20. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  21. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  22. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  23. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  24. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  25. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  26. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  27. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  28. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  29. 2024Generative Representational Instruction Tuningcited by 227
  30. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  31. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  32. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  33. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
  34. 2022Dense Text Retrieval based on Pretrained Language Models: A Surveycited by 136
  35. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  36. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  37. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  38. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 131; 2026: 73.

233 papers.

Sort:

By area:

  1. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  2. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  3. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  4. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  5. 2024Arctic-Embed 2.0: Multilingual Retrieval Without Compromisecited by 57
  6. 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contextscited by 52
  7. 2025Text-to-LoRA: Instant Transformer Adaptioncited by 50
  8. 2024RAGEval: Scenario Specific RAG Evaluation Dataset Generation Frameworkcited by 39
  9. 2025mmBERT: A Modern Multilingual Encoder with Annealed Language Learningcited by 39
  10. 2024jina-clip-v2: Multilingual Multimodal Embeddings for Text and Imagescited by 36
  11. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  12. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 35
38 more
  1. 2024Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance?cited by 34
  2. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  3. 2025LettuceDetect: A Hallucination Detection Framework for RAG Applicationscited by 32
  4. 2024Assessing and Learning Alignment of Unimodal Vision and Language Modelscited by 31
  5. 2025ConTextTab: A Semantics-Aware Tabular In-Context Learnercited by 31
  6. 2024The Automated Verification of Textual Claims (AVeriTeC) Shared Taskcited by 29
  7. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  8. 2025EuroBERT: Scaling Multilingual Encoders for European Languagescited by 28
  9. 2025Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routingcited by 25
  10. 2024Data Extraction Attacks in Retrieval-Augmented Generation via Backdoorscited by 23
  11. 2024When Machine Unlearning Meets Retrieval-Augmented Generation (RAG): Keep Secret or Forget Knowledge?cited by 20
  12. 2024FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answeringcited by 19
  13. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  14. 2026jina-embeddings-v5-text: Task-Targeted Embedding Distillationcited by 19
  15. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 18
  16. 2024MSc-SQL: Multi-Sample Critiquing Small Language Models For Text-To-SQL Translationcited by 17
  17. 2025PyLate: Flexible Training and Retrieval for Late Interaction Modelscited by 17
  18. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  19. 2026Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systemscited by 16
  20. 2024Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialoguecited by 15
  21. 2024HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Predictioncited by 15
  22. 2025vCache: Verified Semantic Prompt Cachingcited by 15
  23. 2024Improving General Text Embedding Model: Tackling Task Conflict and Data Imbalance through Model Mergingcited by 14
  24. 2025NeoBERT: A Next-Generation BERTcited by 14
  25. 2025Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systemscited by 14
  26. 2025LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Modelscited by 13
  27. 2024AIR-Bench: Automated Heterogeneous Information Retrieval Benchmarkcited by 12
  28. 2025Interpretable Text Embeddings and Text Similarity Explanation: A Surveycited by 12
  29. 2025MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluationcited by 11
  30. 2025ValuesRAG: Enhancing Cultural Alignment Through Retrieval-Augmented Contextual Learningcited by 10
  31. 2025FaMTEB: Massive Text Embedding Benchmark in Persian Languagecited by 10
  32. 2025SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversitycited by 10
  33. 2025RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary Quality-Diversity Searchcited by 10
  34. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  35. 2025WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrievalcited by 9
  36. 2025MIRACL-VISION: A Large, multilingual, visual document retrieval benchmarkcited by 9
  37. 2025KG-RAG: Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generationcited by 9
  38. 2024Drowning in Documents: Consequences of Scaling Reranker Inferencecited by 8

50 most cited of 233 All 233 (JSON)