mlsanityData to 7 Oct 2026

Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation

arXiv 2203.09435 ↗First posted 17 Mar 2022cs.CLcited by 44

Not on the map. Why?

Abstract (arXiv)

The performance of multilingual pretrained models is highly dependent on the availability of monolingual or parallel text present in a target language. Thus, the majority of the world's languages cannot benefit from recent progress in NLP as they have no or limited textual data. To expand possibilities of using NLP technology in these under-represented languages, we systematically study strategies that relax the reliance on conventional language resources through the use of bilingual lexicons, an alternative resource with much better language coverage. We analyze different strategies to synthesize textual or labeled data using lexicons, and how this data can be combined with monolingual or parallel text when available. For 19 under-represented languages across 3 tasks, our methods lead to consistent improvements of up to 5 and 15 points with and without extra monolingual text respectively. Overall, our study highlights how NLP methods can be adapted to thousands more languages that are under-served by current technology

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

31 of 49 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  7. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  8. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  9. 2019How multilingual is Multilingual BERT?cited by 642
  10. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  11. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  12. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
19 more
  1. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
  2. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  3. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 142
  4. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  5. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 118
  6. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  7. 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
  8. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  9. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  10. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  11. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  12. 2020OCR Post Correction for Endangered Language Textscited by 31
  13. 2019Pushing the Limits of Low-Resource Morphological Inflectioncited by 29
  14. 2021Lexicon Learning for Few-Shot Neural Sequence Modelingcited by 28
  15. 2016Massively Multilingual Word Embeddingscited by 26
  16. 2021PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretrainingcited by 21
  17. 2021Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Studycited by 19
  18. 2021Modelling Latent Translations for Cross-Lingual Transfercited by 16
  19. 2016Learning Crosslingual Word Embeddings without Bilingual Corporacited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 11; 2024: 16; 2025: 2; 2026: 2.

44 papers.

Sort:

By area:

  1. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  2. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
  3. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  4. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 74
  5. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 67
  6. 2022Language Modelling with Pixelscited by 53
  7. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  8. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  9. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  10. 2022Phylogeny-Inspired Adaptation of Multilingual Models to New Languagescited by 24
  11. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  12. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24
32 more
  1. 2024Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Interventioncited by 24
  2. 2024Exploring Design Choices for Building Language-Specific LLMscited by 22
  3. 2023Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroesecited by 15
  4. 2023Translation and Fusion Improves Zero-shot Cross-lingual Information Extractioncited by 15
  5. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  6. 2024LexC-Gen: Generating Data for Extremely Low-Resource Languages with Large Language Models and Bilingual Lexiconscited by 11
  7. 2023Language Embeddings Sometimes Contain Typological Generalizationscited by 9
  8. 2022Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoderscited by 7
  9. 2022The first neural machine translation system for the Erzya languagecited by 7
  10. 2022Data-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languagescited by 7
  11. 2022Improving Bilingual Lexicon Induction with Cross-Encoder Rerankingcited by 7
  12. 2023Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translationcited by 7
  13. 2024Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexiconcited by 7
  14. 2024How Transliterations Improve Crosslingual Alignmentcited by 6
  15. 2022Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Modelscited by 5
  16. 2023Mitigating Data Imbalance and Representation Degeneration in Multilingual Machine Translationcited by 5
  17. 2022Graph-Based Multilingual Label Propagation for Low-Resource Part-of-Speech Taggingcited by 4
  18. 2024Self-Augmented In-Context Learning for Unsupervised Word Translationcited by 4
  19. 2024Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts?cited by 4
  20. 2023On Bilingual Lexicon Induction with Large Language Modelscited by 2
  21. 2024A Morphologically-Aware Dictionary-based Data Augmentation Technique for Machine Translation of Under-Represented Languagescited by 2
  22. 2024ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Modelcited by 2
  23. 2024LEIA: Facilitating Cross-lingual Knowledge Transfer in Language Models with Entity-based Data Augmentationcited by 1
  24. 2024Incorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfercited by 1
  25. 2024ChatZero:Zero-shot Cross-Lingual Dialogue Generation via Pseudo-Target Languagecited by 1
  26. 2022SilverAlign: MT-Based Silver Data Algorithm For Evaluating Word Alignmentcited by 0
  27. 2022Learnings from Technological Interventions in a Low Resource Language: Enhancing Information Access in Gondicited by 0
  28. 2024Transferring BERT Capabilities from High-Resource to Low-Resource Languages Using Vocabulary Matchingcited by 0
  29. 2024NusaBERT: Teaching IndoBERT to be Multilingual and Multiculturalcited by 0
  30. 2025Adapting Language Models to Indonesian Local Languages: An Empirical Study of Language Transferability on Zero-Shot Settingscited by 0
  31. 2026Effective vocabulary expansion of multilingual language models for extremely low-resource languagescited by 0
  32. 2026Resource-Lean Lexicon Induction for German Dialectscited by 0