mlsanityData to 7 Oct 2026

Adapting BigScience Multilingual Model to Unseen Languages

arXiv 2204.04873 ↗First posted 11 Apr 2022cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

We benchmark different strategies of adding new languages (German and Korean) into the BigScience's pretrained multilingual language model with 1.3 billion parameters that currently supports 13 languages. We investigate the factors that affect the language adaptability of the model and the trade-offs between computational costs and expected performance.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 11 Apr 2022.

Builds on

20 of 32 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  4. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  5. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  6. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  7. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  8. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  9. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  10. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  11. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  12. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
8 more
  1. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  2. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  3. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  4. 2020Multi-Stage Pre-training for Low-Resource Domain Adaptationcited by 24
  5. 2021Multilingual Unsupervised Neural Machine Translation with Denoising Adapterscited by 22
  6. 2021Continual Learning in Multilingual NMT via Language-Specific Embeddingscited by 12
  7. 2021Analyzing the Effects of Reasoning Types on Cross-Lingual Transfer Performancecited by 6
  8. 2021The Impact of Positional Encodings on Multilingual Compressioncited by 5

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 2.

6 papers.

  1. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  2. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  3. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  4. 2024The Hidden Space of Transformer Language Adapterscited by 16
  5. 2023Extending the Pre-Training of BLOOM for Improved Support of Traditional Chinese: Models, Methods and Resultscited by 10
  6. 2024ChocoLlama: Lessons Learned From Teaching Llamas Dutchcited by 5