# Model Selection for Cross-Lingual Transfer

arXiv: https://arxiv.org/abs/2010.06127 · HTML: https://mlsanity.com/paper/2010.06127/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2020.

## Built on by (most cited first, 9 of 9)

- [Frustratingly Easy Label Projection for Cross-lingual Transfer](https://mlsanity.com/paper/2211.15613/) (2022)
- [Zero-shot cross-lingual transfer language selection using linguistic similarity](https://mlsanity.com/paper/2301.13720/) (2023)
- [Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution](https://mlsanity.com/paper/2305.17325/) (2023)
- [Contextual Label Projection for Cross-Lingual Structured Prediction](https://mlsanity.com/paper/2309.08943/) (2023)
- [Constrained Decoding for Cross-lingual Label Projection](https://mlsanity.com/paper/2402.03131/) (2024)
- [Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategy](https://mlsanity.com/paper/2412.09415/) (2024)
- [Meta-Tuning LLMs to Leverage Lexical Knowledge for Generalizable Language Style Understanding](https://mlsanity.com/paper/2305.14592/) (2023)
- [Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptions](https://mlsanity.com/paper/2509.20645/) (2025)
- [Language Distances are Practical for Equitable Cross-Lingual Transfer](https://mlsanity.com/paper/2609.32331/) (2026)

## Abstract (arXiv)

Transformers that are pre-trained on multilingual corpora, such as, mBERT and XLM-RoBERTa, have achieved impressive cross-lingual transfer capabilities. In the zero-shot transfer setting, only English training data is used, and the fine-tuned model is evaluated on another target language. While this works surprisingly well, substantial variance has been observed in target language performance between different fine-tuning runs, and in the zero-shot setup, no target-language development data is available to select among multiple fine-tuned models. Prior work has relied on English dev data to select among models that are fine-tuned with different learning rates, number of steps and other hyperparameters, often resulting in suboptimal choices. In this paper, we show that it is possible to select consistently better models when small amounts of annotated data are available in auxiliary pivot languages. We propose a machine learning approach to model selection that uses the fine-tuned model's own internal representations to predict its cross-lingual capabilities. In extensive experiments we find that this method consistently selects better models than English validation data across twenty five languages (including eight low-resource languages), and often achieves results that are comparable to model selection using target language development data.