mlsanityData to 7 Oct 2026

XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization

arXiv 2003.11080 ↗First posted 24 Mar 2020cs.CL cs.LGcited by 738

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders XTREME benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We release the benchmark to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks.

Key facts. Cited by 738 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 161; 2022: 158; 2023: 137; 2024: 109; 2025: 99; 2026: 74.

738 papers.

Sort:

By area:

  1. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  2. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  3. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  4. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  5. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  6. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  7. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  8. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  9. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  10. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  11. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  12. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
38 more
  1. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  2. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  3. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  4. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  5. 2021MuRIL: Multilingual Representations for Indian Languagescited by 177
  6. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  7. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  8. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  9. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  10. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  11. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  12. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  13. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  14. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  15. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
  16. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  17. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  18. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 142
  19. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  20. 2021The Benchmark Lotterycited by 133
  21. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  22. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  23. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  24. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  25. 2021XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyondcited by 114
  26. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  27. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  28. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  29. 2021Neural Machine Translation for Low-Resource Languages: A Surveycited by 108
  30. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  31. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  32. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  33. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 100
  34. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  35. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  36. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  37. 2022The Geometry of Multilingual Language Model Representationscited by 94
  38. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 94

50 most cited of 738 All 738 (JSON)