mlsanityData to 14 Oct 2026

BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting

arXiv 2212.09535 ↗First posted 19 Dec 2022cs.CL cs.AI cs.LGcited by 90

Not on the map. Why?

Abstract (arXiv)

The BLOOM model is a large publicly available multilingual language model, but its pretraining was limited to 46 languages. To extend the benefits of BLOOM to other languages without incurring prohibitively large costs, it is desirable to adapt BLOOM to new languages not seen during pretraining. In this work, we apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting. We find language adaptation to be effective at improving zero-shot performance in new languages. Surprisingly, we find that adapter-based finetuning is more effective than continued pretraining for large models. In addition, we discover that prompting performance is not significantly affected by language specifics, such as the writing system. It is primarily determined by the size of the language adaptation data. We also add new languages to BLOOMZ, which is a multitask finetuned version of BLOOM capable of following task instructions zero-shot. We find including a new language in the multitask fine-tuning mixture to be the most effective method to teach BLOOMZ a new language. We conclude that with sufficient training data language adaptation can generalize well to diverse languages. Our code is available at https://github.com/bigscience-workshop/multilingual-modeling.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Builds on 55 of its 78 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

55 of 78 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
38 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  2. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  3. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  4. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  6. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  7. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  8. 2019How multilingual is Multilingual BERT?cited by 645
  9. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  10. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  11. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  12. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  13. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  14. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  15. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  16. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  17. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  18. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  19. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  20. 2021How Many Data Points is a Prompt Worth?cited by 198
  21. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  22. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  23. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  24. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  25. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  26. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  27. 2021KLUE: Korean Language Understanding Evaluationcited by 114
  28. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  29. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  30. 2018Rapid Adaptation of Neural Machine Translation to New Languagescited by 82
  31. 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
  32. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  33. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  34. 2021Discrete and Soft Prompting for Multilingual Modelscited by 49
  35. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  36. 2021IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initializationcited by 25
  37. 2020Multi-Stage Pre-training for Low-Resource Domain Adaptationcited by 24
  38. 2022Phylogeny-Inspired Adaptation of Multilingual Models to New Languagescited by 24

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 22; 2024: 35; 2025: 29; 2026: 4.

90 papers.

Sort:

By area:

  1. 2023Scaling Data-Constrained Language Modelscited by 403
  2. 2023Llemma: An Open Language Model For Mathematicscited by 388
  3. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  4. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  5. 2024Generative Representational Instruction Tuningcited by 233
  6. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  7. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  8. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  9. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  10. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 131
  11. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  12. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
38 more
  1. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  2. 2023FinGPT: Large Generative Models for a Small Languagecited by 49
  3. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  4. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 45
  5. 2024Latxa: An Open Language Model and Evaluation Suite for Basquecited by 42
  6. 2024GraphEdit: Large Language Models for Graph Structure Learningcited by 33
  7. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attackscited by 33
  8. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languagescited by 31
  9. 2024Teaching Large Language Models an Unseen Language on the Flycited by 30
  10. 2023Efficiently Adapting Pretrained Language Models To New Languagescited by 28
  11. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  12. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  13. 2023LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Languagecited by 25
  14. 2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 25
  15. 2024Teaching Llama a New Language Through Cross-Lingual Knowledge Transfercited by 23
  16. 2023Using ChatGPT for Entity Matchingcited by 22
  17. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  18. 2023InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuningcited by 21
  19. 2023Sabi'a: Portuguese Large Language Modelscited by 19
  20. 2023Column Type Annotation using ChatGPTcited by 19
  21. 2024Unlocking the Potential of Model Merging for Low-Resource Languagescited by 19
  22. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inferencecited by 18
  23. 2024How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?cited by 13
  24. 2024Small Languages, Big Models: A Study of Continual Training on Languages of Norwaycited by 12
  25. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
  26. 2024LexC-Gen: Generating Data for Extremely Low-Resource Languages with Large Language Models and Bilingual Lexiconscited by 11
  27. 2024SambaLingo: Teaching Large Language Models New Languagescited by 11
  28. 2024MINERS: Multilingual Language Models as Semantic Retrieverscited by 11
  29. 2024Large Vocabulary Size Improves Large Language Modelscited by 11
  30. 2025Beyond Text Compression: Evaluating Tokenizers Across Scalescited by 11
  31. 2023Extending the Pre-Training of BLOOM for Improved Support of Traditional Chinese: Models, Methods and Resultscited by 10
  32. 2023Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognitioncited by 10
  33. 2024LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalizationcited by 10
  34. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  35. 2025Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindicited by 10
  36. 2023CreoleVal: Multilingual Multitask Benchmarks for Creolescited by 9
  37. 2024Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methodscited by 9
  38. 2023A Comprehensive Analysis of Adapter Efficiencycited by 8

50 most cited of 90 All 90 (JSON)