mlsanityData to 7 Oct 2026

Language Models are Few-shot Multilingual Learners

arXiv 2109.07684 ↗First posted 16 Sep 2021cs.CL cs.AIcited by 92

Not on the map. Why?

Abstract (arXiv)

General-purpose language models have demonstrated impressive capabilities, performing on par with state-of-the-art approaches on a range of downstream natural language processing (NLP) tasks and benchmarks when inferring instructions from very few examples. Here, we evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages without any parameter updates. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones. Finally, we find the in-context few-shot cross-lingual prediction results of language models are significantly better than random prediction, and they are competitive compared to the existing state-of-the-art cross-lingual models.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 62 of its 74 references that are in the map. First posted on arXiv 16 Sep 2021.

Builds on

62 of 74 references.

References · 62

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
38 more
  1. 2019Language Models as Knowledge Bases?cited by 1,922
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  5. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  10. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  11. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  12. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  13. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  14. 2019CamemBERT: a Tasty French Language Modelcited by 368
  15. 2021True Few-Shot Learning with Language Modelscited by 343
  16. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  17. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  18. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 156
  19. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  20. 2020PhoBERT: Pre-trained language models for Vietnamesecited by 148
  21. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 143
  22. 2018Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialogcited by 124
  23. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  24. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  25. 2020Knowledge-Grounded Dialogue Generation with Pre-trained Language Modelscited by 107
  26. 2019A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understandingcited by 106
  27. 2020MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systemscited by 103
  28. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  29. 2020End-to-End Slot Alignment and Recognition for Cross-Lingual NLUcited by 82
  30. 2021Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Trackingcited by 61
  31. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  32. 2021Are Multilingual Models Effective in Code-Switching?cited by 56
  33. 2019Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentencescited by 55
  34. 2021BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modelingcited by 55
  35. 2019Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systemscited by 42
  36. 2020Language Models as Few-Shot Learner for Task-Oriented Dialogue Systemscited by 41
  37. 2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
  38. 2020BARThez: a Skilled Pretrained French Sequence-to-Sequence Modelcited by 39

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 22; 2023: 22; 2024: 26; 2025: 13; 2026: 5.

92 papers.

Sort:

By area:

  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  2. 2021Ethical and social risks of harm from Language Modelscited by 957
  3. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  4. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  5. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  6. 2023Co-Writing with Opinionated Language Models Affects Users' Viewscited by 145
  7. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  8. 2024Large Language Models Meet NLP: A Surveycited by 102
  9. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 96
  10. 2023Unifying Molecular and Textual Representations via Multi-task Language Modellingcited by 92
  11. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  12. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
38 more
  1. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  2. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 67
  3. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 65
  4. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  5. 2024Attention Heads of Large Language Models: A Surveycited by 65
  6. 2021Few-Shot Bot: Prompt-Based Learning for Dialogue Systemscited by 64
  7. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 60
  8. 2024Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Modelscited by 51
  9. 2024Fine-tuning Large Language Models for Domain-specific Machine Translationcited by 43
  10. 2022Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extractioncited by 40
  11. 2025Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Studycited by 38
  12. 2022The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challengescited by 35
  13. 2022XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsingcited by 32
  14. 2022Zero-Shot Rumor Detection with Propagation Structure via Prompt Learningcited by 32
  15. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
  16. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  17. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  18. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  19. 2023MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognitioncited by 26
  20. 2022Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Promptcited by 25
  21. 2022QAmeleon: Multilingual QA with Only 5 Examplescited by 25
  22. 2023Fluid Transformers and Creative Analogies: Exploring Large Language Models' Capacity for Augmenting Cross-Domain Analogical Creativitycited by 23
  23. 2024Unveiling Linguistic Regions in Large Language Modelscited by 22
  24. 2025Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMscited by 22
  25. 2022TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languagescited by 21
  26. 2023Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrationscited by 20
  27. 2022SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Taskscited by 19
  28. 2023Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddingscited by 19
  29. 2022Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languagescited by 17
  30. 2022Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Modelscited by 16
  31. 2023CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Modelscited by 16
  32. 2024adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgroundscited by 14
  33. 2024Can we teach language models to gloss endangered languages?cited by 14
  34. 2022TAPE: Assessing Few-shot Russian Language Understandingcited by 11
  35. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  36. 2024MINERS: Multilingual Language Models as Semantic Retrieverscited by 11
  37. 2025Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Modelscited by 11
  38. 2023Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecturecited by 10

50 most cited of 92 All 92 (JSON)