# Phylogeny-Inspired Adaptation of Multilingual Models to New Languages

arXiv: https://arxiv.org/abs/2205.09634 · HTML: https://mlsanity.com/paper/2205.09634/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 42 references that are in the map.
- First posted on arXiv 19 May 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Characterizing and Avoiding Negative Transfer](https://mlsanity.com/paper/1811.09751/) (2018)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://mlsanity.com/paper/2010.12858/) (2020)
- [UDapter: Language Adaptation for Truly Universal Dependency Parsing](https://mlsanity.com/paper/2004.14327/) (2020)
- [AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages](https://mlsanity.com/paper/2104.08726/) (2021)
- [Inducing Language-Agnostic Multilingual Representations](https://mlsanity.com/paper/2008.09112/) (2020)
- [Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation](https://mlsanity.com/paper/2203.09435/) (2022)
- [Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigation](https://mlsanity.com/paper/2008.06788/) (2020)
- [Probing Multilingual BERT for Genetic and Typological Signals](https://mlsanity.com/paper/2011.02070/) (2020)
- [Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability](https://mlsanity.com/paper/2203.10753/) (2022)
- [Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models](https://mlsanity.com/paper/2205.02023/) (2022)
- [Adaptable Adapters](https://mlsanity.com/paper/2205.01549/) (2022)

## Built on by (most cited first, 24 of 24)

- [BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting](https://mlsanity.com/paper/2212.09535/) (2022)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)](https://mlsanity.com/paper/2304.06845/) (2023)
- [Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models](https://mlsanity.com/paper/2401.10440/) (2024)
- [MaLA-500: Massive Language Adaptation of Large Language Models](https://mlsanity.com/paper/2401.13303/) (2024)
- [Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages](https://mlsanity.com/paper/2309.04679/) (2023)
- [Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroese](https://mlsanity.com/paper/2304.08823/) (2023)
- [Cross-Lingual Transfer with Target Language-Ready Task Adapters](https://mlsanity.com/paper/2306.02767/) (2023)
- [BIG-C: a Multimodal Multi-Purpose Dataset for Bemba](https://mlsanity.com/paper/2305.17202/) (2023)
- [Targeted Multilingual Adaptation for Low-resource Language Families](https://mlsanity.com/paper/2405.12413/) (2024)
- [An Efficient Approach for Studying Cross-Lingual Transfer in Multilingual Language Models](https://mlsanity.com/paper/2403.20088/) (2024)
- [LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages](https://mlsanity.com/paper/2305.14263/) (2023)
- [The Impact of Language Adapters in Cross-Lingual Transfer for NLU](https://mlsanity.com/paper/2402.00149/) (2024)
- [To token or not to token: A Comparative Study of Text Representations for Cross-Lingual Transfer](https://mlsanity.com/paper/2310.08078/) (2023)
- [Adapting to the Low-Resource Double-Bind: Investigating Low-Compute Methods on Low-Resource African Languages](https://mlsanity.com/paper/2303.16985/) (2023)
- [Data-Augmentation-Based Dialectal Adaptation for LLMs](https://mlsanity.com/paper/2404.08092/) (2024)
- [Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages](https://mlsanity.com/paper/2502.10140/) (2025)
- [Modality Matching Matters: Calibrating Language Distances for Cross-Lingual Transfer in URIEL+](https://mlsanity.com/paper/2510.19217/) (2025)
- [GMNLP at SemEval-2023 Task 12: Sentiment Analysis with Phylogeny-Based Adapters](https://mlsanity.com/paper/2304.12979/) (2023)
- [Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT](https://mlsanity.com/paper/2412.12674/) (2024)
- [Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models](https://mlsanity.com/paper/2510.13580/) (2025)
- [How to Tune a Multilingual Encoder Model for Germanic Languages: A Study of PEFT, Full Fine-Tuning, and Language Adapters](https://mlsanity.com/paper/2501.06025/) (2025)
- [UoB-NLP at SemEval-2025 Task 11: Leveraging Adapters for Multilingual and Cross-Lingual Emotion Detection](https://mlsanity.com/paper/2504.08543/) (2025)
- [Limited-Resource Adapters Are Regularizers, Not Linguists](https://mlsanity.com/paper/2505.24525/) (2025)

## Abstract (arXiv)

Large pretrained multilingual models, trained on dozens of languages, have delivered promising results due to cross-lingual learning capabilities on variety of language tasks. Further adapting these models to specific languages, especially ones unseen during pre-training, is an important goal towards expanding the coverage of language technologies. In this study, we show how we can use language phylogenetic information to improve cross-lingual transfer leveraging closely related languages in a structured, linguistically-informed manner. We perform adapter-based training on languages from diverse language families (Germanic, Uralic, Tupian, Uto-Aztecan) and evaluate on both syntactic and semantic tasks, obtaining more than 20% relative performance improvements over strong commonly used baselines, especially on languages unseen during pre-training.