mlsanityData to 14 Oct 2026

Specializing Multilingual Language Models: An Empirical Study

arXiv 2106.09063 ↗First posted 16 Jun 2021cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Pretrained multilingual language models have become a common tool in transferring NLP capabilities to low-resource languages, often with adaptations. In this work, we study the performance, extensibility, and interaction of two such adaptations: vocabulary augmentation and script transliteration. Our evaluations on part-of-speech tagging, universal dependency parsing, and named entity recognition in nine diverse low-resource languages uphold the viability of these approaches while raising new questions around how to optimally adapt multilingual models to low-resource settings.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 55 of its 72 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

55 of 72 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2018Deep contextualized word representationscited by 2,290
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  12. 2019Cross-lingual Language Model Pretrainingcited by 1,090
38 more
  1. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  2. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  3. 2019How multilingual is Multilingual BERT?cited by 645
  4. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  5. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  6. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  7. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  8. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  9. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  10. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  11. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  12. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  13. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  14. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  15. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  16. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  17. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  18. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  19. 2020Named Entity Recognition as Dependency Parsingcited by 151
  20. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  21. 2020Multilingual Alignment of Contextual Word Representationscited by 114
  22. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  23. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  24. 2015Multi-task Sequence to Sequence Learningcited by 94
  25. 2017Adversarial Multi-task Learning for Text Classificationcited by 91
  26. 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
  27. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  28. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  29. 2016A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Taskscited by 62
  30. 2018A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Taskscited by 54
  31. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  32. 2021Multi-view Subword Regularizationcited by 40
  33. 2020Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfercited by 23
  34. 2018Scheduled Multi-Task Learning: From Syntax to Translationcited by 18
  35. 2018Multi-Task Learning for Sequence Tagging: An Empirical Studycited by 15
  36. 2021PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and dependency parsingcited by 13
  37. 2018Multitask Parsing Across Semantic Representationscited by 10
  38. 2019Adversarial Multitask Learning for Joint Multi-Feature and Multi-Dialect Morphological Modelingcited by 9

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 4; 2024: 6; 2025: 2; 2026: 3.

21 papers.

Sort:

By area:

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  2. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  3. 2022Language Modelling with Pixelscited by 55
  4. 2024RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanizationcited by 29
  5. 2023Romanization-based Large-scale Adaptation of Multilingual Language Modelscited by 21
  6. 2022Improving Word Translation via Two-Stage Contrastive Learningcited by 18
  7. 2022Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltesecited by 14
  8. 2022MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learningcited by 12
  9. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languagescited by 12
  10. 2024TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Datacited by 9
  11. 2022Exploring Dimensionality Reduction Techniques in Multilingual Transformerscited by 6
  12. 2024Beyond Data Quantity: Key Factors Driving Performance in Multilingual Language Modelscited by 6
9 more
  1. 2024Comparing LLM prompting with Cross-lingual transfer performance on Indigenous and Low-resource Brazilian Languagescited by 4
  2. 2024Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignmentcited by 2
  3. 2025An evaluation of LLMs and Google Translate for translation of selected Indian languages via sentiment and semantic analysescited by 2
  4. 2023PrOnto: Language Model Evaluations for 859 Languagescited by 1
  5. 2024Training a Bilingual Language Model by Mapping Tokens onto a Shared Character Spacecited by 1
  6. 2025What Really Counts? Examining Step and Token Level Attribution in Multilingual CoT Reasoningcited by 1
  7. 2023FewShotTextGCN: K-hop neighborhood regularization for few-shot learning on graphscited by 0
  8. 2026Modular Monolingual Adaptation using Pretrained Language Modelscited by 0
  9. 2026Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinyacited by 0