mlsanityData to 13 Oct 2026

Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval

arXiv 2108.08787 ↗First posted 19 Aug 2021cs.CL cs.IRcited by 119

Not on the map. Why?

Abstract (arXiv)

We present Mr. TyDi, a multi-lingual benchmark dataset for mono-lingual retrieval in eleven typologically diverse languages, designed to evaluate ranking with learned dense representations. The goal of this resource is to spur research in dense retrieval techniques in non-English languages, motivated by recent observations that existing techniques for representation learning perform poorly when applied to out-of-distribution data. As a starting point, we provide zero-shot baselines for this new dataset based on a multi-lingual adaptation of DPR that we call "mDPR". Experiments show that although the effectiveness of mDPR is much lower than BM25, dense representations nevertheless appear to provide valuable relevance signals, improving BM25 results in sparse-dense hybrids. In addition to analyses of our results, we also discuss future challenges and present a research agenda in multi-lingual dense retrieval. Mr. TyDi can be downloaded at https://github.com/castorini/mr.tydi.

Key facts. Cited by 119 papers in the map (data through 13 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 19 Aug 2021.

Builds on

18 of 24 references.

References · 18

  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,797
  3. 2017Billion-scale similarity search with GPUscited by 2,490
  4. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,600
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,237
  6. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,169
  7. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 847
  8. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  9. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 487
  10. 2020Overview of the TREC 2019 deep learning trackcited by 285
  11. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  12. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
6 more
  1. 2021Overview of the TREC 2020 deep learning trackcited by 195
  2. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 152
  3. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 144
  4. 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
  5. 2021A Replication Study of Dense Passage Retrievercited by 37
  6. 2020Complementing Lexical Retrieval with Semantic Residual Embeddingcited by 27

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 5; 2022: 15; 2023: 18; 2024: 34; 2025: 24; 2026: 23.

119 papers.

Sort:

By area:

  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,205
  2. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,058
  3. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 510
  4. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 467
  5. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 398
  6. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 364
  7. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 358
  8. 2023Improving Text Embeddings with Large Language Modelscited by 297
  9. 2022RealTime QA: What's the Answer Right Now?cited by 242
  10. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 235
  11. 2024Generative Representational Instruction Tuningcited by 229
  12. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 205
38 more
  1. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  2. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  3. 2024Making Text Embedders Few-Shot Learnerscited by 79
  4. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 77
  5. 2024Repetition Improves Language Model Embeddingscited by 71
  6. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  7. 2025Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Taskscited by 48
  8. 2022Tevatron: An Efficient and Flexible Toolkit for Dense Retrievalcited by 47
  9. 2022Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languagescited by 47
  10. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 35
  11. 2022XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsingcited by 33
  12. 2021Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representationscited by 27
  13. 2024AutoRAG: Automated Framework for optimization of Retrieval Augmented Generation Pipelinecited by 27
  14. 2022Towards Best Practices for Training Multilingual Dense Retrieval Modelscited by 26
  15. 2024Position: Measure Dataset Diversity, Don't Just Claim Itcited by 26
  16. 2024Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retrievercited by 25
  17. 2023Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrievalcited by 24
  18. 2021On Cross-Lingual Retrieval with Multilingual Text Encoderscited by 22
  19. 2024The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model designcited by 20
  20. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmarkcited by 19
  21. 2024Pooling And Attention: What Are Effective Designs For LLM-Based Embedding Models?cited by 19
  22. 2022Neural Approaches to Multilingual Information Retrievalcited by 18
  23. 2023GAR-meets-RAG Paradigm for Zero-Shot Information Retrievalcited by 18
  24. 2023BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Languagecited by 15
  25. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  26. 2024PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methodscited by 14
  27. 2024AIR-Bench: Automated Heterogeneous Information Retrieval Benchmarkcited by 12
  28. 2026F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual Worldcited by 12
  29. 2022MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languagescited by 11
  30. 2023Language Models are Universal Embedderscited by 11
  31. 2022HC4: A New Suite of Test Collections for Ad Hoc CLIRcited by 10
  32. 2023Soft Prompt Decoding for Multilingual Dense Retrievalcited by 10
  33. 2024Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarkscited by 10
  34. 2025WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrievalcited by 10
  35. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  36. 2025Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connectorcited by 9
  37. 2021Encoder Adaptation of Dense Passage Retrieval for Open-Domain Question Answeringcited by 8
  38. 2022Language Agnostic Multilingual Information Retrieval with Contrastive Learningcited by 8

50 most cited of 119 All 119 (JSON)