# MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning

arXiv: https://arxiv.org/abs/2104.07908 · HTML: https://mlsanity.com/paper/2104.07908/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Meta Pseudo Labels](https://mlsanity.com/paper/2003.10580/) (2020)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [The Multilingual Amazon Reviews Corpus](https://mlsanity.com/paper/2010.02573/) (2020)
- [Controllable Invariance through Adversarial Feature Learning](https://mlsanity.com/paper/1705.11122/) (2017)
- [Meta Label Correction for Noisy Label Learning](https://mlsanity.com/paper/1911.03809/) (2019)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1808.08437/) (2018)
- [Generalized Data Augmentation for Low-Resource Translation](https://mlsanity.com/paper/1906.03785/) (2019)
- [Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification](https://mlsanity.com/paper/1606.01614/) (2016)
- [Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks](https://mlsanity.com/paper/1908.10423/) (2019)
- [A Call for More Rigor in Unsupervised Cross-lingual Learning](https://mlsanity.com/paper/2004.14958/) (2020)
- [Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces](https://mlsanity.com/paper/1908.06625/) (2019)
- [SentiPers: A Sentiment Analysis Corpus for Persian](https://mlsanity.com/paper/1801.07737/) (2018)
- [Towards Zero-resource Cross-lingual Entity Linking](https://mlsanity.com/paper/1909.13180/) (2019)

## Built on by (most cited first, 21 of 21)

- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis](https://mlsanity.com/paper/2201.08277/) (2022)
- [Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error](https://mlsanity.com/paper/2106.09613/) (2021)
- [Meta Learning for Natural Language Processing: A Survey](https://mlsanity.com/paper/2205.01500/) (2022)
- [MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialogue](https://mlsanity.com/paper/2212.10455/) (2022)
- [The Zeno's Paradox of `Low-Resource' Languages](https://mlsanity.com/paper/2410.20817/) (2024)
- [Meta-Learning a Cross-lingual Manifold for Semantic Parsing](https://mlsanity.com/paper/2209.12577/) (2022)
- [A Dataset and Baselines for Multilingual Reply Suggestion](https://mlsanity.com/paper/2106.02017/) (2021)
- [EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization](https://mlsanity.com/paper/2106.10575/) (2021)
- [A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space](https://mlsanity.com/paper/2109.06324/) (2021)
- [CrossAligner & Co: Zero-Shot Transfer Methods for Task-Oriented Cross-lingual Natural Language Understanding](https://mlsanity.com/paper/2203.09982/) (2022)
- [PromptDA: Label-guided Data Augmentation for Prompt-based Few-shot Learners](https://mlsanity.com/paper/2205.09229/) (2022)
- [Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexicon](https://mlsanity.com/paper/2402.02113/) (2024)
- [Know Where You're Going: Meta-Learning for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2205.12453/) (2022)
- [CREST: Universal Safety Guardrails Through Cluster-Guided Cross-Lingual Transfer](https://mlsanity.com/paper/2512.02711/) (2025)
- [Constrained Density Matching and Modeling for Cross-lingual Alignment of Contextualized Representations](https://mlsanity.com/paper/2201.13429/) (2022)
- [MetaGAD: Meta Representation Adaptation for Few-Shot Graph Anomaly Detection](https://mlsanity.com/paper/2305.10668/) (2023)
- [MetaXT: Meta Cross-Task Transfer between Disparate Label Spaces](https://mlsanity.com/paper/2109.04240/) (2021)
- [Persian Natural Language Inference: A Meta-learning approach](https://mlsanity.com/paper/2205.08755/) (2022)
- [MetaXLR -- Mixed Language Meta Representation Transformation for Low-resource Cross-lingual Learning based on Multi-Armed Bandit](https://mlsanity.com/paper/2306.00100/) (2023)
- [Towards Cross-Lingual Audio Abuse Detection in Low-Resource Settings with Few-Shot Learning](https://mlsanity.com/paper/2412.01408/) (2024)

## Abstract (arXiv)

The combination of multilingual pre-trained representations and cross-lingual transfer learning is one of the most effective methods for building functional NLP systems for low-resource languages. However, for extremely low-resource languages without large-scale monolingual corpora for pre-training or sufficient annotated data for fine-tuning, transfer learning remains an under-studied and challenging task. Moreover, recent work shows that multilingual representations are surprisingly disjoint across languages, bringing additional challenges for transfer onto extremely low-resource languages. In this paper, we propose MetaXL, a meta-learning based framework that learns to transform representations judiciously from auxiliary languages to a target one and brings their representation spaces closer for effective transfer. Extensive experiments on real-world low-resource languages - without access to large-scale monolingual corpora or large amounts of labeled data - for tasks like cross-lingual sentiment analysis and named entity recognition show the effectiveness of our approach. Code for MetaXL is publicly available at github.com/microsoft/MetaXL.