# ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapters

arXiv: https://arxiv.org/abs/2310.16393 · HTML: https://mlsanity.com/paper/2310.16393/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [MasakhaNER: Named Entity Recognition for African Languages](https://mlsanity.com/paper/2103.11811/) (2021)
- [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://mlsanity.com/paper/2106.03164/) (2021)
- [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://mlsanity.com/paper/2010.12858/) (2020)
- [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://mlsanity.com/paper/2012.15562/) (2020)
- [UDapter: Language Adaptation for Truly Universal Dependency Parsing](https://mlsanity.com/paper/2004.14327/) (2020)
- [Contextual Parameter Generation for Universal Neural Machine Translation](https://mlsanity.com/paper/1808.08493/) (2018)
- [KinyaBERT: a Morphology-aware Kinyarwanda Language Model](https://mlsanity.com/paper/2203.08459/) (2022)
- [Efficient Test Time Adapter Ensembling for Low-resource Language Varieties](https://mlsanity.com/paper/2109.04877/) (2021)
- [mOKB6: A Multilingual Open Knowledge Base Completion Benchmark](https://mlsanity.com/paper/2211.06959/) (2022)
- [Multilingual Fact Linking](https://mlsanity.com/paper/2109.14364/) (2021)
- [PARE: A Simple and Strong Baseline for Monolingual and Multilingual Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2110.07415/) (2021)

## Built on by (most cited first, 10 of 10)

- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalization](https://mlsanity.com/paper/2401.06034/) (2024)
- [A Three-Pronged Approach to Cross-Lingual Adaptation with Multilingual LLMs](https://mlsanity.com/paper/2406.17377/) (2024)
- [Cross-lingual Text Classification Transfer: The Case of Ukrainian](https://mlsanity.com/paper/2404.02043/) (2024)
- [Soft Language Prompts for Language Transfer](https://mlsanity.com/paper/2407.02317/) (2024)
- [SSP: Self-Supervised Prompting for Cross-Lingual Transfer to Low-Resource Languages using Large Language Models](https://mlsanity.com/paper/2406.18880/) (2024)
- [Cross-Lingual IPA Contrastive Learning for Zero-Shot NER](https://mlsanity.com/paper/2503.07214/) (2025)
- [Zero-Shot Cross-Lingual NER Using Phonemic Representations for Low-Resource Languages](https://mlsanity.com/paper/2406.16030/) (2024)
- [Limited-Resource Adapters Are Regularizers, Not Linguists](https://mlsanity.com/paper/2505.24525/) (2025)
- [Harnessing Linguistic Dissimilarity for Language Generalization on Unseen Low-Resource Varieties](https://mlsanity.com/paper/2605.04500/) (2026)

## Abstract (arXiv)

We tackle the problem of zero-shot cross-lingual transfer in NLP tasks via the use of language adapters (LAs). Most of the earlier works have explored training with adapter of a single source (often English), and testing either using the target LA or LA of another related language. Training target LA requires unlabeled data, which may not be readily available for low resource unseen languages: those that are neither seen by the underlying multilingual language model (e.g., mBERT), nor do we have any (labeled or unlabeled) data for them. We posit that for more effective cross-lingual transfer, instead of just one source LA, we need to leverage LAs of multiple (linguistically or geographically related) source languages, both at train and test-time - which we investigate via our novel neural architecture, ZGUL. Extensive experimentation across four language groups, covering 15 unseen target languages, demonstrates improvements of up to 3.2 average F1 points over standard fine-tuning and other strong baselines on POS tagging and NER tasks. We also extend ZGUL to settings where either (1) some unlabeled data or (2) few-shot training examples are available for the target language. We find that ZGUL continues to outperform baselines in these settings too.