mlsanityData to 7 Oct 2026

Predicting the Performance of Multilingual NLP Models

arXiv 2110.08875 ↗First posted 17 Oct 2021cs.CL cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Recent advancements in NLP have given us models like mBERT and XLMR that can serve over 100 languages. The languages that these models are evaluated on, however, are very few in number, and it is unlikely that evaluation datasets will cover all the languages that these models support. Potential solutions to the costly problem of dataset creation are to translate datasets to new languages or use template-filling based techniques for creation. This paper proposes an alternate solution for evaluating a model across languages which make use of the existing performance scores of the model on languages that a particular task has test sets for. We train a predictor on these performance scores and use this predictor to predict the model's performance in different evaluation settings. Our results show that our method is effective in filling the gaps in the evaluation for an existing set of languages, but might require additional improvements if we want it to generalize to unseen languages.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 33 of its 39 references that are in the map. First posted on arXiv 17 Oct 2021.

Builds on

33 of 39 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  9. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  10. 2019How multilingual is Multilingual BERT?cited by 642
  11. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  12. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
21 more
  1. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  2. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  3. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  4. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  5. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  6. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  7. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 143
  8. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  9. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  10. 2019Multilingual is not enough: BERT for Finnishcited by 118
  11. 2019BERTje: A Dutch BERT Modelcited by 109
  12. 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 101
  13. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  14. 2020LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluationcited by 92
  15. 2017Transfer Learning across Low-Resource, Related Languages for Neural Machine Translationcited by 58
  16. 2020Predicting Performance for Natural Language Processing Taskscited by 49
  17. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  18. 2020A Call for More Rigor in Unsupervised Cross-lingual Learningcited by 37
  19. 2021Towards More Fine-grained and Reliable NLP Performance Predictioncited by 26
  20. 2020WikiBERT models: deep transfer learning for many languagescited by 20
  21. 2019Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communitiescited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 3; 2024: 8; 2025: 6; 2026: 1.

24 papers.

Sort:

  1. 2023Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Reviewcited by 50
  2. 2024URIEL+: Enhancing Linguistic Inclusion and Usability in a Typological and Multilingual Knowledge Basecited by 21
  3. 2022Multi Task Learning For Zero Shot Performance Prediction of Multilingual Modelscited by 20
  4. 2024ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Modelscited by 20
  5. 2022On the Calibration of Massively Multilingual Language Modelscited by 18
  6. 2023Cultural Compass: Predicting Transfer Learning Success in Offensive Language Detection with Cultural Featurescited by 18
  7. 2023Unsupervised Layer-wise Score Aggregation for Textual OOD Detectioncited by 15
  8. 2022Predicting Fine-Tuning Performance with Probingcited by 13
  9. 2024Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications?cited by 12
  10. 2022Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languagescited by 10
  11. 2025Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Mattercited by 10
  12. 2024A Reproducibility Study on Quantifying Language Similarity: The Impact of Missing Values in the URIEL Knowledge Basecited by 7
12 more
  1. 2024On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?cited by 5
  2. 2024Decoding the Diversity: A Review of the Indic AI Research Landscapecited by 4
  3. 2025SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methodscited by 4
  4. 2022On the Economics of Multilingual Few-shot Learning: Modeling the Cost-Performance Trade-offs of Machine Translated and Manual Datacited by 3
  5. 2025Modality Matching Matters: Calibrating Language Distances for Cross-Lingual Transfer in URIEL+cited by 3
  6. 2022Detecting Languages Unintelligible to Multilingual Models through Local Structure Probescited by 2
  7. 2025Zero-Shot Performance Prediction for Probabilistic Scaling Lawscited by 2
  8. 2025Anticipatory Evaluation of Language Modelscited by 1
  9. 2024Evaluation Methodology for Large Language Models for Multilingual Document Question and Answercited by 0
  10. 2024Zero-shot Cross-lingual Transfer Learning with Multiple Source and Target Languages for Information Extraction: Language Selection and Adversarial Trainingcited by 0
  11. 2025Multilinguality Does not Make Sense: Investigating Factors Behind Zero-Shot Transfer in Sense-Aware Taskscited by 0
  12. 2026Language Distances are Practical for Equitable Cross-Lingual Transfercited by 0