# LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalization

arXiv: https://arxiv.org/abs/2401.06034 · HTML: https://mlsanity.com/paper/2401.06034/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 61 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://mlsanity.com/paper/2012.15562/) (2020)
- [Yara Parser: A Fast and Accurate Dependency Parser](https://mlsanity.com/paper/1503.06733/) (2015)
- [IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding](https://mlsanity.com/paper/2009.05387/) (2020)
- [Compressing Large-Scale Transformer-Based Models: A Case Study on BERT](https://mlsanity.com/paper/2002.11985/) (2020)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2105.10311/) (2021)
- [Stance Detection with Bidirectional Conditional Encoding](https://mlsanity.com/paper/1606.05464/) (2016)
- [BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting](https://mlsanity.com/paper/2212.09535/) (2022)
- [Larger-Scale Transformers for Multilingual Masked Language Modeling](https://mlsanity.com/paper/2105.00572/) (2021)
- [UDapter: Language Adaptation for Truly Universal Dependency Parsing](https://mlsanity.com/paper/2004.14327/) (2020)
- [Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing](https://mlsanity.com/paper/1807.00914/) (2018)
- [IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation](https://mlsanity.com/paper/2104.08200/) (2021)
- [Multilingual Neural Machine Translation with Language Clustering](https://mlsanity.com/paper/1908.09324/) (2019)
- [Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations](https://mlsanity.com/paper/2004.14923/) (2020)
- [ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapters](https://mlsanity.com/paper/2310.16393/) (2023)
- [The State of NLP Literature: A Diachronic Analysis of the ACL Anthology](https://mlsanity.com/paper/1911.03562/) (2019)
- [Massive-scale Decoding for Text Generation using Lattices](https://mlsanity.com/paper/2112.07660/) (2021)

## Built on by (most cited first, 10 of 10)

- [LLMs Are Few-Shot In-Context Low-Resource Language Learners](https://mlsanity.com/paper/2403.16512/) (2024)
- [URIEL+: Enhancing Linguistic Inclusion and Usability in a Typological and Multilingual Knowledge Base](https://mlsanity.com/paper/2409.18472/) (2024)
- [ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Models](https://mlsanity.com/paper/2406.09334/) (2024)
- [A Reproducibility Study on Quantifying Language Similarity: The Impact of Missing Values in the URIEL Knowledge Base](https://mlsanity.com/paper/2405.11125/) (2024)
- [NusaAksara: A Multimodal and Multilingual Benchmark for Preserving Indonesian Indigenous Scripts](https://mlsanity.com/paper/2502.18148/) (2025)
- [Modality Matching Matters: Calibrating Language Distances for Cross-Lingual Transfer in URIEL+](https://mlsanity.com/paper/2510.19217/) (2025)
- [Entropy2Vec: Crosslingual Language Modeling Entropy as End-to-End Learnable Language Representations](https://mlsanity.com/paper/2509.05060/) (2025)
- [Harnessing Linguistic Dissimilarity for Language Generalization on Unseen Low-Resource Varieties](https://mlsanity.com/paper/2605.04500/) (2026)
- [Self-Supervised Lexical Representation Learning for Fast, Large-Scale Phylogenetic Inference](https://mlsanity.com/paper/2609.05262/) (2026)
- [Language Distances are Practical for Equitable Cross-Lingual Transfer](https://mlsanity.com/paper/2609.32331/) (2026)

## Abstract (arXiv)

Pretrained language models (PLMs) have become remarkably adept at task and language generalization. Nonetheless, they often fail when faced with unseen languages. In this work, we present LinguAlchemy, a regularization method that incorporates various linguistic information covering typological, geographical, and phylogenetic features to align PLMs representation to the corresponding linguistic information on each language. Our LinguAlchemy significantly improves the performance of mBERT and XLM-R on low-resource languages in multiple downstream tasks such as intent classification, news classification, and semantic relatedness compared to fully finetuned models and displaying a high degree of unseen language generalization. We further introduce AlchemyScale and AlchemyTune, extension of LinguAlchemy which adjusts the linguistic regularization weights automatically, alleviating the need for hyperparameter search.