mlsanityData to 7 Oct 2026

Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languages

arXiv 2205.06356 ↗First posted 12 May 2022cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Although recent Massively Multilingual Language Models (MMLMs) like mBERT and XLMR support around 100 languages, most existing multilingual NLP benchmarks provide evaluation data in only a handful of these languages with little linguistic diversity. We argue that this makes the existing practices in multilingual evaluation unreliable and does not provide a full picture of the performance of MMLMs across the linguistic landscape. We propose that the recent work done in Performance Prediction for NLP tasks can serve as a potential solution in fixing benchmarking in Multilingual NLP by utilizing features related to data and language typology to estimate the performance of an MMLM on different languages. We compare performance prediction with translating test data with a case study on four different multilingual datasets, and observe that these methods can provide reliable estimates of the performance that are often on-par with the translation based approaches, without the need for any additional translation as well as evaluation costs.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 12 May 2022.

Builds on

36 of 48 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  6. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  7. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  8. 2019How multilingual is Multilingual BERT?cited by 642
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  10. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  11. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  12. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
24 more
  1. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  2. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  3. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  4. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  5. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  6. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  7. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  8. 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 182
  9. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  10. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  11. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  12. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 143
  13. 2019Transferability and Hardness of Supervised Classification Taskscited by 125
  14. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  15. 2020Entity Linking in 100 Languagescited by 52
  16. 2020Predicting Performance for Natural Language Processing Taskscited by 49
  17. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  18. 2021From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understandingcited by 43
  19. 2018A Corpus for Multilingual Document Classification in Eight Languagescited by 42
  20. 2020LAReQA: Language-agnostic answer retrieval from a multilingual poolcited by 41
  21. 2022QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakerscited by 39
  22. 2021Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linkingcited by 27
  23. 2021Towards More Fine-grained and Reliable NLP Performance Predictioncited by 26
  24. 2021Predicting the Performance of Multilingual NLP Modelscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 2; 2025: 3; 2026: 2.

10 papers.

  1. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  2. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  3. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  4. 2022On the Calibration of Massively Multilingual Language Modelscited by 18
  5. 2024METAL: Towards Multilingual Meta-Evaluationcited by 14
  6. 2026Translation as a Scalable Proxy for Multilingual Evaluationcited by 6
  7. 2025Uncovering inequalities in new knowledge learning by large language models across different languagescited by 1
  8. 2025FAME: Fictional Actors for Multilingual Erasurecited by 1
  9. 2025SMAB: MAB based word Sensitivity Estimation Framework and its Applications in Adversarial Text Generationcited by 0
  10. 2026Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Modelscited by 0