The Less the Merrier? Investigating Language Representation in Multilingual Models
Not on the map. Why?
Abstract (arXiv)
Multilingual Language Models offer a way to incorporate multiple languages in one model and utilize cross-language transfer learning to improve performance for different Natural Language Processing (NLP) tasks. Despite progress in multilingual models, not all languages are supported as well, particularly in low-resource settings. In this work, we investigate the linguistic representation of different languages in multilingual models. We start by asking the question which languages are supported in popular multilingual models and which languages are left behind. Then, for included languages, we look at models' learned representations based on language family and dialect and try to understand how models' learned representations for~(1) seen and~(2) unseen languages vary across different language groups. In addition, we test and analyze performance on downstream tasks such as text generation and Named Entity Recognition. We observe from our experiments that community-centered models -- models that focus on languages of a given family or geographical location and are built by communities who speak them -- perform better at distinguishing between languages in the same family for low-resource languages. Our paper contributes to the literature in understanding multilingual models and their shortcomings and offers insights on potential ways to improve them.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 20 Oct 2023.
Builds on
29 of 43 references.
References · 29
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
- 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
- 2019How multilingual is Multilingual BERT?cited by 642
- 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
- 2018Characterizing and Avoiding Negative Transfercited by 226
17 more
- 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
- 2018Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialogcited by 124
- 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
- 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
- 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
- 2022The Geometry of Multilingual Language Model Representationscited by 94
- 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
- 2019How Language-Neutral is Multilingual BERT?cited by 51
- 2021A Primer on Pretrained Multilingual Language Modelscited by 49
- 2022AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languagescited by 46
- 2021IndicBART: A Pre-trained Model for Indic Natural Language Generationcited by 39
- 2021An Isotropy Analysis in the Multilingual BERT Embedding Spacecited by 25
- 2020What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological propertiescited by 23
- 2023MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languagescited by 23
- 2020AMBERT: A Pre-trained Language Model with Multi-Grained Tokenizationcited by 22
- 2018Natural Language Processing for Information Extractioncited by 14
- 2023GPT-SW3: An Autoregressive Language Model for the Nordic Languagescited by 8
Built on by
Citing papers per year: 2025: 2.
2 papers.