mlsanityData to 14 Oct 2026

mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset

arXiv 2108.13897 ↗First posted 31 Aug 2021cs.CL cs.AIcited by 116

Not on the map. Why?

Abstract (arXiv)

The MS MARCO ranking dataset has been widely used for training deep learning models for IR tasks, achieving considerable effectiveness on diverse zero-shot scenarios. However, this type of resource is scarce in languages other than English. In this work, we present mMARCO, a multilingual version of the MS MARCO passage ranking dataset comprising 13 languages that was created using machine translation. We evaluated mMARCO by finetuning monolingual and multilingual reranking models, as well as a multilingual dense retrieval model on this dataset. We also evaluated models finetuned using the mMARCO dataset in a zero-shot scenario on Mr. TyDi dataset, demonstrating that multilingual models finetuned on our translated dataset achieve superior effectiveness to models finetuned on the original English version alone. Our experiments also show that a distilled multilingual reranker is competitive with non-distilled models while having 5.4 times fewer parameters. Lastly, we show a positive correlation between translation quality and retrieval effectiveness, providing evidence that improvements in translation methods might lead to improvements in multilingual information retrieval. The translated datasets and finetuned models are available at https://github.com/unicamp-dl/mMARCO.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). Builds on 43 of its 52 references that are in the map. First posted on arXiv 31 Aug 2021.

Builds on

43 of 52 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  7. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  8. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  9. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  10. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  12. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
31 more
  1. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  2. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  3. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  4. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  5. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  6. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  7. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  8. 2020Overview of the TREC 2019 deep learning trackcited by 286
  9. 2018Marian: Fast Neural Machine Translation in C++cited by 203
  10. 2019Multi-Stage Document Ranking with BERTcited by 201
  11. 2021Overview of the TREC 2020 deep learning trackcited by 196
  12. 2013Exploiting Similarities among Languages for Machine Translationcited by 191
  13. 2020A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarioscited by 171
  14. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 145
  15. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 119
  16. 2020Syntactic Data Augmentation Increases Robustness to Inference Heuristicscited by 86
  17. 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
  18. 2020The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizescited by 48
  19. 2020RepBERT: Contextualized Text Embeddings for First-Stage Retrievalcited by 46
  20. 2020PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrievalcited by 40
  21. 2019Automatic Spanish Translation of the SQuAD Dataset for Multilingual Question Answeringcited by 35
  22. 2020PTT5: Pretraining and validating the T5 model on Brazilian Portuguese datacited by 35
  23. 2016Massively Multilingual Word Embeddingscited by 26
  24. 2020Expansion via Prediction of Importance with Contextualizationcited by 26
  25. 2020TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrievalcited by 24
  26. 2021Should we Stop Training More Monolingual Models, and Simply Use Machine Translation Instead?cited by 17
  27. 2018Unsupervised Multilingual Word Embeddingscited by 16
  28. 2019Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-shot Learningcited by 16
  29. 2020Understanding BERT Rankers Under Distillationcited by 15
  30. 2021A cost-benefit analysis of cross-lingual transfer methodscited by 7
  31. 2020On the Reliability of Test Collections for Evaluating Systems of Different Typescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 11; 2023: 19; 2024: 31; 2025: 25; 2026: 29.

116 papers.

Sort:

By area:

  1. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  2. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  3. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  4. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  5. 2023Zero-Shot Listwise Document Reranking with a Large Language Modelcited by 82
  6. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 66
  7. 2022Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languagescited by 47
  8. 2022Transfer Learning Approaches for Building Cross-Language Dense Retrieval Modelscited by 36
  9. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 36
  10. 2024The Chronicles of RAG: The Retriever, the Chunk and the Generatorcited by 34
  11. 2022Towards Best Practices for Training Multilingual Dense Retrieval Modelscited by 26
  12. 2024Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retrievercited by 25
38 more
  1. 2023Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrievalcited by 24
  2. 2024INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languagescited by 24
  3. 2025jina-reranker-v3: Last but Not Late Interaction for Listwise Document Rerankingcited by 24
  4. 2024Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillationcited by 22
  5. 2022Neural Approaches to Multilingual Information Retrievalcited by 18
  6. 2023GAR-meets-RAG Paradigm for Zero-Shot Information Retrievalcited by 18
  7. 2022Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrievalcited by 17
  8. 2022RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Questioncited by 17
  9. 2024LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Rankingcited by 16
  10. 2022DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Enginecited by 15
  11. 2023Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillationcited by 15
  12. 2023BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Languagecited by 15
  13. 2024JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create State-of-the-Art Japanese Retrievers with Constrained Resourcescited by 15
  14. 2026F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual Worldcited by 12
  15. 2023MAUPQA: Massive Automatically-created Polish Question Answering Datasetcited by 11
  16. 2023Language Models are Universal Embedderscited by 11
  17. 2023Soft Prompt Decoding for Multilingual Dense Retrievalcited by 10
  18. 2024Piccolo2: General Text Embedding with Multi-task Hybrid Loss Trainingcited by 10
  19. 2024Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarkscited by 10
  20. 2025FaMTEB: Massive Text Embedding Benchmark in Persian Languagecited by 10
  21. 2023Finding the Law: Enhancing Statutory Article Retrieval via Graph Neural Networkscited by 9
  22. 2024Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Modelscited by 9
  23. 2025Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connectorcited by 9
  24. 2024ptt5-v2: A Closer Look at Continued Pretraining of T5 Models for the Portuguese Languagecited by 8
  25. 2024Cross-Dialect Information Retrieval: Information Access in Low-Resource and High-Variance Languagescited by 8
  26. 2024ColBERT-XM: A Modular Multi-Vector Representation Model for Zero-Shot Multilingual Information Retrievalcited by 7
  27. 2024KazQAD: Kazakh Open-Domain Question Answering Datasetcited by 7
  28. 2025mFollowIR: a Multilingual Benchmark for Instruction Following in Retrievalcited by 7
  29. 2024BoolQuestions: Does Dense Retrieval Understand Boolean Logic in Language?cited by 6
  30. 2024BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Languagecited by 6
  31. 2026DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Searchcited by 6
  32. 2023Silver Retriever: Advancing Neural Passage Retrieval for Polish Question Answeringcited by 5
  33. 2023Multi-EuP: The Multilingual European Parliament Dataset for Analysis of Bias in Information Retrievalcited by 5
  34. 2023IndicIRSuite: Multilingual Dataset and Neural Information Models for Indian Languagescited by 5
  35. 2025MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutchcited by 5
  36. 2025M3Retrieve: Benchmarking Multimodal Retrieval for Medicinecited by 5
  37. 2022Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval with Adapterscited by 4
  38. 2023Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrievalcited by 4

50 most cited of 116 All 116 (JSON)