mlsanityData to 14 Oct 2026

Few-shot Learning with Multilingual Language Models

arXiv 2112.10668 ↗First posted 20 Dec 2021cs.CL cs.AIcited by 278

Not on the map. Why?

Abstract (arXiv)

Large-scale generative language models such as GPT-3 are competitive few-shot learners. While these models are known to be able to jointly represent many different languages, their training data is dominated by English, potentially limiting their cross-lingual generalization. In this work, we train multilingual generative language models on a corpus covering a diverse set of languages, and study their few- and zero-shot learning capabilities in a wide range of tasks. Our largest model with 7.5 billion parameters sets new state of the art in few-shot learning in more than 20 representative languages, outperforming GPT-3 of comparable size in multilingual commonsense reasoning (with +7.4% absolute accuracy improvement in 0-shot settings and +9.4% in 4-shot settings) and natural language inference (+5.4% in each of 0-shot and 4-shot settings). On the FLORES-101 machine translation benchmark, our model outperforms GPT-3 on 171 out of 182 directions with 32 training examples, while surpassing the official supervised baseline in 45 directions. We conduct an in-depth analysis of different multilingual prompting approaches, showing in particular that strong few-shot learning performance across languages can be achieved via cross-lingual transfer through both templates and demonstration examples. Finally, we evaluate our models in social value tasks such as hate speech detection in five languages and find it has limitations similar to comparable sized GPT-3 models.

Key facts. Cited by 278 papers in the map (data through 14 Oct 2026). Builds on 56 of its 77 references that are in the map. First posted on arXiv 20 Dec 2021.

Builds on

56 of 77 references.

References · 56

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2019Language Models as Knowledge Bases?cited by 1,936
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  13. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  14. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  15. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  16. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  17. 2020Aligning AI With Shared Human Valuescited by 723
  18. 2021GPT Understands, Toocited by 716
  19. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  20. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  21. 2019How multilingual is Multilingual BERT?cited by 645
  22. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  23. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  24. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  25. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
  26. 2021True Few-Shot Learning with Language Modelscited by 344
  27. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  28. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 315
  29. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  30. 2018A Simple Method for Commonsense Reasoningcited by 223
  31. 2013Exploiting Similarities among Languages for Machine Translationcited by 191
  32. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  33. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  34. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  35. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  36. 2020Translation Artifacts in Cross-lingual Transfer Learningcited by 96
  37. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  38. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 87

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 76; 2024: 134; 2025: 47; 2026: 12.

278 papers.

Sort:

By area:

  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  3. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  4. 2023The Falcon Series of Open Language Modelscited by 535
  5. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  6. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  7. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  8. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  9. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
  10. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
  11. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  12. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
38 more
  1. 2024Investigating Cultural Alignment of Large Language Modelscited by 143
  2. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  3. 2023Hallucinations in Large Multilingual Translation Modelscited by 129
  4. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  5. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 127
  6. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  7. 2023Lift Yourself Up: Retrieval-augmented Text Generation with Self Memorycited by 103
  8. 2024Large Language Models Meet NLP: A Surveycited by 103
  9. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 103
  10. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  11. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 97
  12. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  13. 2023Do Multilingual Language Models Think Better in English?cited by 88
  14. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 88
  15. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 86
  16. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 84
  17. 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
  18. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 81
  19. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  20. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  21. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  22. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  23. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
  24. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  25. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  26. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 59
  27. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  28. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 56
  29. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  30. 2023TIM: Teaching Large Language Models to Translate with Comparisoncited by 50
  31. 2022Bidirectional Language Models Are Also Few-shot Learnerscited by 48
  32. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
  33. 2023Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Promptscited by 45
  34. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  35. 2024Fine-tuning Large Language Models for Domain-specific Machine Translationcited by 43
  36. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 43
  37. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 43
  38. 2024How Does Quantization Affect Multilingual LLMs?cited by 43

50 most cited of 278 All 278 (JSON)