# Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models

arXiv: https://arxiv.org/abs/2205.06130 · HTML: https://mlsanity.com/paper/2205.06130/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 54 references that are in the map.
- First posted on arXiv 12 May 2022.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019)
- [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [Entity Linking in 100 Languages](https://mlsanity.com/paper/2011.02690/) (2020)
- [Predicting Performance for Natural Language Processing Tasks](https://mlsanity.com/paper/2005.00870/) (2020)
- [A Corpus for Multilingual Document Classification in Eight Languages](https://mlsanity.com/paper/1805.09821/) (2018)
- [LAReQA: Language-agnostic answer retrieval from a multilingual pool](https://mlsanity.com/paper/2004.05484/) (2020)
- [Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approach](https://mlsanity.com/paper/1705.07086/) (2017)
- [Towards More Fine-grained and Reliable NLP Performance Prediction](https://mlsanity.com/paper/2102.05486/) (2021)
- [Predicting the Performance of Multilingual NLP Models](https://mlsanity.com/paper/2110.08875/) (2021)
- [Analysing The Impact Of Linguistic Features On Cross-Lingual Transfer](https://mlsanity.com/paper/2105.05975/) (2021)
- [Cold-start recommendations in Collective Matrix Factorization](https://mlsanity.com/paper/1809.00366/) (2018)

## Built on by (most cited first, 20 of 20)

- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review](https://mlsanity.com/paper/2305.16768/) (2023)
- [MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition](https://mlsanity.com/paper/2210.12391/) (2022)
- [Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languages](https://mlsanity.com/paper/2205.12676/) (2022)
- [Cultural Compass: Predicting Transfer Learning Success in Offensive Language Detection with Cultural Features](https://mlsanity.com/paper/2310.06458/) (2023)
- [TaTa: A Multilingual Table-to-Text Dataset for African Languages](https://mlsanity.com/paper/2211.00142/) (2022)
- [Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter](https://mlsanity.com/paper/2501.14491/) (2025)
- [Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment](https://mlsanity.com/paper/2508.08424/) (2025)
- [Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization](https://mlsanity.com/paper/2406.13718/) (2024)
- [Cross-lingual Transfer in Programming Languages: An Extensive Empirical Study](https://mlsanity.com/paper/2310.16937/) (2023)
- [On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?](https://mlsanity.com/paper/2406.14267/) (2024)
- [Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning](https://mlsanity.com/paper/2210.06068/) (2022)
- [Modality Matching Matters: Calibrating Language Distances for Cross-Lingual Transfer in URIEL+](https://mlsanity.com/paper/2510.19217/) (2025)
- [Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes](https://mlsanity.com/paper/2211.05015/) (2022)
- [Overview of the First Workshop on Language Models for Low-Resource Languages (LoResLM 2025)](https://mlsanity.com/paper/2412.16365/) (2024)
- [Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions](https://mlsanity.com/paper/2509.20645/) (2025)
- [Dialetto, ma Quanto Dialetto? Transcribing and Evaluating Dialects on a Continuum](https://mlsanity.com/paper/2410.14589/) (2024)
- [Assessing the Role of Lexical Semantics in Cross-lingual Transfer through Controlled Manipulations](https://mlsanity.com/paper/2408.07599/) (2024)
- [SMAB: MAB based word Sensitivity Estimation Framework and its Applications in Adversarial Text Generation](https://mlsanity.com/paper/2502.07101/) (2025)

## Abstract (arXiv)

Massively Multilingual Transformer based Language Models have been observed to be surprisingly effective on zero-shot transfer across languages, though the performance varies from language to language depending on the pivot language(s) used for fine-tuning. In this work, we build upon some of the existing techniques for predicting the zero-shot performance on a task, by modeling it as a multi-task learning problem. We jointly train predictive models for different tasks which helps us build more accurate predictors for tasks where we have test data in very few languages to measure the actual performance of the model. Our approach also lends us the ability to perform a much more robust feature selection and identify a common set of features that influence zero-shot performance across a variety of tasks.