mlsanityData to 7 Oct 2026

The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation

arXiv 2106.03193 ↗First posted 6 Jun 2021cs.CL cs.AIcited by 556

Not on the map. Why?

Abstract (arXiv)

One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource languages, consider only restricted domains, or are low quality because they are constructed using semi-automatic procedures. In this work, we introduce the FLORES-101 evaluation benchmark, consisting of 3001 sentences extracted from English Wikipedia and covering a variety of different topics and domains. These sentences have been translated in 101 languages by professional translators through a carefully controlled process. The resulting dataset enables better assessment of model quality on the long tail of low-resource languages, including the evaluation of many-to-many multilingual translation systems, as all translations are multilingually aligned. By publicly releasing such a high-quality and high-coverage dataset, we hope to foster progress in the machine translation community and beyond.

Key facts. Cited by 556 papers in the map (data through 7 Oct 2026). Builds on 37 of its 66 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

37 of 66 references.

References · 37

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  6. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  7. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  8. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  9. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  10. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  11. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  12. 2019Massively Multilingual Neural Machine Translationcited by 246
25 more
  1. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  2. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  3. 2020Participatory Research for Low-resourced Machine Translation: A Case Study in African Languagescited by 159
  4. 2018Phrase-Based & Neural Unsupervised Machine Translationcited by 138
  5. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  6. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 127
  7. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  8. 2020TICO-19: the Translation Initiative for Covid-19cited by 67
  9. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 61
  10. 2018Unsupervised Statistical Machine Translationcited by 51
  11. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  12. 2019The Effect of Translationese in Machine Translation Test Setscited by 49
  13. 2017Learning Language Representations for Typology Predictioncited by 40
  14. 2020Complete Multilingual Neural Machine Translationcited by 30
  15. 2020FFR v1.1: Fon-French Neural Machine Translationcited by 16
  16. 2020Igbo-English Machine Translation: An Evaluation Benchmarkcited by 14
  17. 2021Extended Parallel Corpus for Amharic-English Machine Translationcited by 10
  18. 2021IndT5: A Text-to-Text Transformer for 10 Indigenous Languagescited by 10
  19. 2018Emerging Language Spaces Learned From Massively Multilingual Corporacited by 9
  20. 2019Benchmarking Neural Machine Translation for Southern African Languagescited by 9
  21. 2021Low-Resource Neural Machine Translation for Southern African Languagescited by 7
  22. 2021Crowdsourced Phrase-Based Tokenization for Low-Resourced Neural Machine Translation: The Case of Fon Languagecited by 6
  23. 2021Towards a parallel corpus of Portuguese and the Bantu language Emakhuwa of Mozambiquecited by 5
  24. 2021Unsupervised Machine Translation On Dravidian Languagescited by 4
  25. 2021Domain-specific MT for Low-resource Languages: The case of Bambara-Frenchcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 64; 2023: 101; 2024: 144; 2025: 131; 2026: 100.

556 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2025Gemma 3 Technical Reportcited by 1,686
  5. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  6. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  7. 2023A Comprehensive Overview of Large Language Modelscited by 639
  8. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  9. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  10. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  11. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  12. 2021Few-shot Learning with Multilingual Language Modelscited by 278
38 more
  1. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  2. 2024Many-Shot In-Context Learningcited by 203
  3. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  4. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  5. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  6. 2022In-context Examples Selection for Machine Translationcited by 172
  7. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  8. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  9. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  10. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 142
  11. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  12. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  13. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  14. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  15. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  16. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
  17. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  18. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  19. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  20. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 97
  21. 2021Survey of Low-Resource Machine Translationcited by 92
  22. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 92
  23. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  24. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  25. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  26. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  27. 2023Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translationcited by 74
  28. 2023YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpuscited by 74
  29. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 74
  30. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  31. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  32. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  33. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  34. 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
  35. 2021The paradox of the compositionality of natural language: a neural machine translation case studycited by 58
  36. 2023Adaptive Machine Translation with Large Language Modelscited by 58
  37. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 58
  38. 2022AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Modelcited by 57

50 most cited of 556 All 556 (JSON)