# Multilingual and cross-lingual document classification: A meta-learning approach

arXiv: https://arxiv.org/abs/2101.11302 · HTML: https://mlsanity.com/paper/2101.11302/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 27 Jan 2021.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples](https://mlsanity.com/paper/1903.03096/) (2019)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning](https://mlsanity.com/paper/1911.04623/) (2019)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1808.08437/) (2018)
- [Gradient Centralization: A New Optimization Technique for Deep Neural Networks](https://mlsanity.com/paper/2004.01461/) (2020)
- [Meta Relational Learning for Few-Shot Link Prediction in Knowledge Graphs](https://mlsanity.com/paper/1909.01515/) (2019)
- [Zero-Shot Cross-Lingual Transfer with Meta Learning](https://mlsanity.com/paper/2003.02739/) (2020)
- [Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks](https://mlsanity.com/paper/1911.03863/) (2019)
- [Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks](https://mlsanity.com/paper/1908.10423/) (2019)
- [A Corpus for Multilingual Document Classification in Eight Languages](https://mlsanity.com/paper/1805.09821/) (2018)
- [MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://mlsanity.com/paper/1909.04761/) (2019)
- [Bridging the domain gap in cross-lingual document classification](https://mlsanity.com/paper/1909.07009/) (2019)
- [A Comparison of Architectures and Pretraining Methods for Contextualized Multilingual Word Embeddings](https://mlsanity.com/paper/1912.10169/) (2019)

## Built on by (most cited first, 14 of 14)

- [Zero-shot cross-lingual transfer language selection using linguistic similarity](https://mlsanity.com/paper/2301.13720/) (2023)
- [Meta Learning for Natural Language Processing: A Survey](https://mlsanity.com/paper/2205.01500/) (2022)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Transfer Language Selection for Zero-Shot Cross-Lingual Abusive Language Detection](https://mlsanity.com/paper/2206.00962/) (2022)
- [A Survey of Event Causality Identification: Taxonomy, Challenges, Assessment, and Prospects](https://mlsanity.com/paper/2411.10371/) (2024)
- [Meta-XNLG: A Meta-Learning Approach Based on Language Clustering for Zero-Shot Cross-Lingual Transfer and Generation](https://mlsanity.com/paper/2203.10250/) (2022)
- [Generalized Funnelling: Ensemble Learning and Heterogeneous Document Embeddings for Cross-Lingual Text Classification](https://mlsanity.com/paper/2110.14764/) (2021)
- [Measuring Catastrophic Forgetting in Cross-Lingual Transfer Paradigms: Exploring Tuning Strategies](https://mlsanity.com/paper/2309.06089/) (2023)
- [Multilingual Sentence-Level Semantic Search using Meta-Distillation Learning](https://mlsanity.com/paper/2309.08185/) (2023)
- [A (More) Realistic Evaluation Setup for Generalisation of Community Models on Malicious Content Detection](https://mlsanity.com/paper/2404.01822/) (2024)
- [Persian Natural Language Inference: A Meta-learning approach](https://mlsanity.com/paper/2205.08755/) (2022)
- [Learning New Tasks from a Few Examples with Soft-Label Prototypes](https://mlsanity.com/paper/2210.17437/) (2022)
- [Towards Cross-Lingual Audio Abuse Detection in Low-Resource Settings with Few-Shot Learning](https://mlsanity.com/paper/2412.01408/) (2024)
- [MAPLE: A Meta-learning Framework for Cross-Prompt Essay Scoring](https://mlsanity.com/paper/2604.17569/) (2026)

## Abstract (arXiv)

The great majority of languages in the world are considered under-resourced for the successful application of deep learning methods. In this work, we propose a meta-learning approach to document classification in limited-resource setting and demonstrate its effectiveness in two different settings: few-shot, cross-lingual adaptation to previously unseen languages; and multilingual joint training when limited target-language data is available during training. We conduct a systematic comparison of several meta-learning methods, investigate multiple settings in terms of data availability and show that meta-learning thrives in settings with a heterogeneous task distribution. We propose a simple, yet effective adjustment to existing meta-learning methods which allows for better and more stable learning, and set a new state of the art on several languages while performing on-par on others, using only a small amount of labeled data.