mlsanityData to 7 Oct 2026

LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalization

arXiv 2401.06034 ↗First posted 11 Jan 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Pretrained language models (PLMs) have become remarkably adept at task and language generalization. Nonetheless, they often fail when faced with unseen languages. In this work, we present LinguAlchemy, a regularization method that incorporates various linguistic information covering typological, geographical, and phylogenetic features to align PLMs representation to the corresponding linguistic information on each language. Our LinguAlchemy significantly improves the performance of mBERT and XLM-R on low-resource languages in multiple downstream tasks such as intent classification, news classification, and semantic relatedness compared to fully finetuned models and displaying a high degree of unseen language generalization. We further introduce AlchemyScale and AlchemyTune, extension of LinguAlchemy which adjusts the linguistic regularization weights automatically, alleviating the need for hyperparameter search.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 31 of its 61 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

31 of 61 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  8. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  9. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  10. 2019CamemBERT: a Tasty French Language Modelcited by 368
  11. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  12. 2017A Structured Self-attentive Sentence Embeddingcited by 318
19 more
  1. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  2. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 143
  3. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  4. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
  5. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  6. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
  7. 2021Pretrained Language Models for Text Generation: A Surveycited by 95
  8. 2016Stance Detection with Bidirectional Conditional Encodingcited by 93
  9. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  10. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  11. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  12. 2018Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processingcited by 63
  13. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  14. 2019Multilingual Neural Machine Translation with Language Clusteringcited by 47
  15. 2020Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representationscited by 21
  16. 2023ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapterscited by 10
  17. 2019The State of NLP Literature: A Diachronic Analysis of the ACL Anthologycited by 8
  18. 2021Massive-scale Decoding for Text Generation using Latticescited by 8
  19. 2023The Obscure Limitation of Modular Multilingual Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 3.

10 papers.

  1. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  2. 2024URIEL+: Enhancing Linguistic Inclusion and Usability in a Typological and Multilingual Knowledge Basecited by 21
  3. 2024ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Modelscited by 20
  4. 2024A Reproducibility Study on Quantifying Language Similarity: The Impact of Missing Values in the URIEL Knowledge Basecited by 7
  5. 2025NusaAksara: A Multimodal and Multilingual Benchmark for Preserving Indonesian Indigenous Scriptscited by 4
  6. 2025Modality Matching Matters: Calibrating Language Distances for Cross-Lingual Transfer in URIEL+cited by 3
  7. 2025Entropy2Vec: Crosslingual Language Modeling Entropy as End-to-End Learnable Language Representationscited by 0
  8. 2026Harnessing Linguistic Dissimilarity for Language Generalization on Unseen Low-Resource Varietiescited by 0
  9. 2026Self-Supervised Lexical Representation Learning for Fast, Large-Scale Phylogenetic Inferencecited by 0
  10. 2026Language Distances are Practical for Equitable Cross-Lingual Transfercited by 0