# Can Monolingual Pretrained Models Help Cross-Lingual Classification?

arXiv: https://arxiv.org/abs/1911.03913 · HTML: https://mlsanity.com/paper/1911.03913/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2019.

## Built on by (most cited first, 6 of 6)

- [Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Models](https://mlsanity.com/paper/2204.08110/) (2022)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [On the ability of monolingual models to learn language-agnostic representations](https://mlsanity.com/paper/2109.01942/) (2021)
- [On the Universality of Deep Contextual Language Models](https://mlsanity.com/paper/2109.07140/) (2021)
- [Cross-Lingual Text Classification with Multilingual Distillation and Zero-Shot-Aware Training](https://mlsanity.com/paper/2202.13654/) (2022)
- [Zero-shot Cross-lingual Transfer without Parallel Corpus](https://mlsanity.com/paper/2310.04726/) (2023)

## Abstract (arXiv)

Multilingual pretrained language models (such as multilingual BERT) have achieved impressive results for cross-lingual transfer. However, due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. In this work, we present two approaches to improve zero-shot cross-lingual classification, by transferring the knowledge from monolingual pretrained models to multilingual ones. Experimental results on two cross-lingual classification benchmarks show that our methods outperform vanilla multilingual fine-tuning.