mlsanityData to 7 Oct 2026

XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation

arXiv 2104.07412 ↗First posted 15 Apr 2021cs.CL cs.AIcited by 167

Not on the map. Why?

Abstract (arXiv)

Machine learning has brought striking advances in multilingual natural language processing capabilities over the past year. For example, the latest techniques have improved the state-of-the-art performance on the XTREME multilingual benchmark by more than 13 points. While a sizeable gap to human-level performance remains, improvements have been easier to achieve in some tasks than in others. This paper analyzes the current state of cross-lingual transfer learning and summarizes some lessons learned. In order to catalyze meaningful progress, we extend XTREME to XTREME-R, which consists of an improved set of ten natural language understanding tasks, including challenging language-agnostic retrieval tasks, and covers 50 typologically diverse languages. In addition, we provide a massively multilingual diagnostic suite (MultiCheckList) and fine-grained multi-dataset evaluation capabilities through an interactive public leaderboard to gain a better understanding of such models. The leaderboard and code for XTREME-R will be made available at https://sites.research.google/xtreme and https://github.com/google-research/xtreme respectively.

Key facts. Cited by 167 papers in the map (data through 7 Oct 2026). Builds on 52 of its 66 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

52 of 66 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
38 more
  1. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  4. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  5. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  6. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  7. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  8. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  9. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  10. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  11. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  12. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  13. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  14. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  15. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  16. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  17. 2020Participatory Research for Low-resourced Machine Translation: A Case Study in African Languagescited by 159
  18. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  19. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  20. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  21. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  22. 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 101
  23. 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
  24. 2018On the Limitations of Unsupervised Bilingual Dictionary Inductioncited by 61
  25. 2020On Learning Universal Representations Across Languagescited by 58
  26. 2020Gender Bias in Multilingual Embeddings and Cross-Lingual Transfercited by 55
  27. 2020Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languagescited by 52
  28. 2020Entity Linking in 100 Languagescited by 52
  29. 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50
  30. 2018End-to-End Retrieval in Continuous Spacecited by 46
  31. 2020ParsiNLU: A Suite of Language Understanding Challenges for Persiancited by 42
  32. 2020LAReQA: Language-agnostic answer retrieval from a multilingual poolcited by 41
  33. 2020English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Toocited by 39
  34. 2020A Call for More Rigor in Unsupervised Cross-lingual Learningcited by 37
  35. 2020From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformerscited by 35
  36. 2020FILTER: An Enhanced Fusion Method for Cross-lingual Language Understandingcited by 29
  37. 2020Utility is in the Eye of the User: A Critique of NLP Leaderboardscited by 28
  38. 2020Interpretable Multi-dataset Evaluation for Named Entity Recognitioncited by 27

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 47; 2023: 33; 2024: 36; 2025: 21; 2026: 12.

167 papers.

Sort:

By area:

  1. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  2. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  3. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  4. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  5. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  6. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  7. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  8. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  9. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  10. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  11. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  12. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
38 more
  1. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  2. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  3. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 94
  4. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 89
  5. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 87
  6. 2023Modular Deep Learningcited by 82
  7. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  8. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluationcited by 73
  9. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  10. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  11. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  12. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  13. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  14. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLUcited by 53
  15. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  16. 2024DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languagescited by 49
  17. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  18. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  19. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
  20. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  21. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 46
  22. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  23. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  24. 2022Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extractioncited by 40
  25. 2023Revisiting Machine Translation for Cross-lingual Classificationcited by 40
  26. 2023PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuningcited by 39
  27. 2022MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognitioncited by 38
  28. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  29. 2022The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challengescited by 35
  30. 2023XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languagescited by 34
  31. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
  32. 2025Training Sparse Mixture Of Experts Text Embedding Modelscited by 32
  33. 2022Enhancing Cross-lingual Transfer by Manifold Mixupcited by 30
  34. 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29
  35. 2022Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languagescited by 28
  36. 2023Zero-shot cross-lingual transfer language selection using linguistic similaritycited by 28
  37. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  38. 2022IndicXNLI: Evaluating Multilingual Inference for Indian Languagescited by 26

50 most cited of 167 All 167 (JSON)