# From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers

arXiv: https://arxiv.org/abs/2005.00633 · HTML: https://mlsanity.com/paper/2005.00633/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2020.

## Built on by (most cited first, 30 of 35)

- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Multi-lingual and Multi-cultural Figurative Language Understanding](https://mlsanity.com/paper/2305.16171/) (2023)
- [Attention Can Reflect Syntactic Structure (If You Let It)](https://mlsanity.com/paper/2101.10927/) (2021)
- [Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer](https://mlsanity.com/paper/2205.12148/) (2022)
- [Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languages](https://mlsanity.com/paper/2205.12676/) (2022)
- [Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages](https://mlsanity.com/paper/2212.09651/) (2022)
- [X-SCITLDR: Cross-Lingual Extreme Summarization of Scholarly Documents](https://mlsanity.com/paper/2205.15051/) (2022)
- [Aligning LLMs for Multilingual Consistency in Enterprise Applications](https://mlsanity.com/paper/2509.23659/) (2025)
- [Meta-Learning for Fast Cross-Lingual Adaptation in Dependency Parsing](https://mlsanity.com/paper/2104.04736/) (2021)
- [Nearest Neighbour Few-Shot Learning for Cross-lingual Classification](https://mlsanity.com/paper/2109.02221/) (2021)
- [Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders](https://mlsanity.com/paper/2511.10840/) (2025)
- [Linguistically inspired roadmap for building biologically reliable protein language models](https://mlsanity.com/paper/2207.00982/) (2022)
- [A Reproducibility Study on Quantifying Language Similarity: The Impact of Missing Values in the URIEL Knowledge Base](https://mlsanity.com/paper/2405.11125/) (2024)
- [Mind the Language Gap: Automated and Augmented Evaluation of Bias in LLMs for High- and Low-Resource Languages](https://mlsanity.com/paper/2504.18560/) (2025)
- [Hate Speech Detection in Limited Data Contexts using Synthetic Data Generation](https://mlsanity.com/paper/2310.02876/) (2023)
- [Can Embedding Similarity Predict Cross-Lingual Transfer? A Systematic Study on African Languages](https://mlsanity.com/paper/2601.03168/) (2026)
- [Cross-Lingual Transfer Learning for Phrase Break Prediction with Multilingual Language Model](https://mlsanity.com/paper/2306.02579/) (2023)
- [A Quantitative Approach to Understand Self-Supervised Models as Cross-lingual Feature Extractors](https://mlsanity.com/paper/2311.15954/) (2023)
- [Automatic Identification of Motivation for Code-Switching in Speech Transcripts](https://mlsanity.com/paper/2212.08565/) (2022)
- [Linguistically-Informed Neural Architectures for Lexical, Syntactic and Semantic Tasks in Sanskrit](https://mlsanity.com/paper/2308.08807/) (2023)
- [DeMuX: Data-efficient Multilingual Learning](https://mlsanity.com/paper/2311.06379/) (2023)
- [SynDARin: Synthesising Datasets for Automated Reasoning in Low-Resource Languages](https://mlsanity.com/paper/2406.14425/) (2024)
- [Learning the Topic, Not the Language: How LLMs Classify Online Immigration Discourse Across Languages](https://mlsanity.com/paper/2508.06435/) (2025)
- [Cross-Lingual Stability and Bias in Instruction-Tuned Language Models for Humanitarian NLP](https://mlsanity.com/paper/2510.22823/) (2025)
- [Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring](https://mlsanity.com/paper/2605.00754/) (2026)
- [Mixture of Experts for Low-Resource LLMs](https://mlsanity.com/paper/2605.17598/) (2026)
- [Self-Attentive Constituency Parsing for UCCA-based Semantic Parsing](https://mlsanity.com/paper/2110.00621/) (2021)
- [Analysing the Greek Parliament Records with Emotion Classification](https://mlsanity.com/paper/2205.12012/) (2022)

## Abstract (arXiv)

Massively multilingual transformers pretrained with language modeling objectives (e.g., mBERT, XLM-R) have become a de facto default transfer paradigm for zero-shot cross-lingual transfer in NLP, offering unmatched transfer performance. Current downstream evaluations, however, verify their efficacy predominantly in transfer settings involving languages with sufficient amounts of pretraining data, and with lexically and typologically close languages. In this work, we analyze their limitations and show that cross-lingual transfer via massively multilingual transformers, much like transfer via cross-lingual word embeddings, is substantially less effective in resource-lean scenarios and for distant languages. Our experiments, encompassing three lower-level tasks (POS tagging, dependency parsing, NER), as well as two high-level semantic tasks (NLI, QA), empirically correlate transfer performance with linguistic similarity between the source and target languages, but also with the size of pretraining corpora of target languages. We also demonstrate a surprising effectiveness of inexpensive few-shot transfer (i.e., fine-tuning on a few target-language instances after fine-tuning in the source) across the board. This suggests that additional research efforts should be invested to reach beyond the limiting zero-shot conditions.