mlsanityData to 7 Oct 2026

AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model

arXiv 2208.01448 ↗First posted 2 Aug 2022cs.CL cs.LGcited by 57

Not on the map. Why?

Abstract (arXiv)

In this work, we demonstrate that multilingual large-scale sequence-to-sequence (seq2seq) models, pre-trained on a mixture of denoising and Causal Language Modeling (CLM) tasks, are more efficient few-shot learners than decoder-only models on various tasks. In particular, we train a 20 billion parameter multilingual seq2seq model called Alexa Teacher Model (AlexaTM 20B) and show that it achieves state-of-the-art (SOTA) performance on 1-shot summarization tasks, outperforming a much larger 540B PaLM decoder model. AlexaTM 20B also achieves SOTA in 1-shot machine translation, especially for low-resource languages, across almost all language pairs supported by the model (Arabic, English, French, German, Hindi, Italian, Japanese, Marathi, Portuguese, Spanish, Tamil, and Telugu) on Flores-101 dataset. We also show in zero-shot setting, AlexaTM 20B outperforms GPT3 (175B) on SuperGLUE and SQuADv2 datasets and provides SOTA performance on multilingual tasks such as XNLI, XCOPA, Paws-X, and XWinograd. Overall, our results present a compelling case for seq2seq models as a powerful alternative to decoder-only models for Large-scale Language Model (LLM) training.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 69 of its 91 references that are in the map. First posted on arXiv 2 Aug 2022.

Builds on

69 of 91 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  13. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  14. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  15. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  16. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  17. 2020On Layer Normalization in the Transformer Architecturecited by 906
  18. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  19. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  20. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  21. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  22. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  23. 2020Recipes for building an open-domain chatbotcited by 593
  24. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  25. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  26. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  27. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  28. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  29. 2018Gender Bias in Coreference Resolutioncited by 438
  30. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  31. 2016Solving General Arithmetic Word Problemscited by 353
  32. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  33. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  34. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  35. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  36. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  37. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  38. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 33; 2024: 12; 2025: 2; 2026: 2.

57 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2023BloombergGPT: A Large Language Model for Financecited by 923
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  6. 2024Large Language Models: A Surveycited by 506
  7. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  8. 2023Scaling Data-Constrained Language Modelscited by 391
  9. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  10. 2023Challenges and Applications of Large Language Modelscited by 251
  11. 2023Efficient Large Language Models: A Surveycited by 183
  12. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
38 more
  1. 2022In-context Examples Selection for Machine Translationcited by 172
  2. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  3. 2023Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answeringcited by 119
  4. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  5. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 58
  6. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  7. 2022Bidirectional Language Models Are Also Few-shot Learnerscited by 48
  8. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  9. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  10. 2023Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuningcited by 40
  11. 2023A Survey on Knowledge Editing of Neural Networkscited by 34
  12. 2024Large Language Models in Cybersecurity: State-of-the-Artcited by 34
  13. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  14. 2022Weakly Supervised Data Augmentation Through Prompting for Dialogue Understandingcited by 32
  15. 2024History, Development, and Principles of Large Language Models-An Introductory Surveycited by 30
  16. 2022LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Taggingcited by 29
  17. 2024Beyond Text: Frozen Large Language Models in Visual Signal Comprehensioncited by 23
  18. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  19. 2022CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsingcited by 21
  20. 2023Transformer models: an introduction and catalogcited by 21
  21. 2023User Simulation with Large Language Models for Evaluating Task-Oriented Dialoguecited by 19
  22. 2023Controlling Personality Style in Dialogue with Zero-Shot Prompt-Based Learningcited by 14
  23. 2023OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model Pre-trained from Scratchcited by 7
  24. 2024Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Studycited by 7
  25. 2023Several categories of Large Language Models (LLMs): A Short Surveycited by 6
  26. 2024A Survey of Large Language Models for European Languagescited by 6
  27. 2022Stateful Memory-Augmented Transformers for Efficient Dialogue Modelingcited by 5
  28. 2025Pensez: Less Data, Better Reasoning -- Rethinking French LLMcited by 5
  29. 2023bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmarkcited by 4
  30. 2023Controllable Generation of Dialogue Acts for Dialogue Systems via Few-Shot Response Generation and Rankingcited by 4
  31. 2023Holistic Survey of Privacy and Fairness in Machine Learningcited by 4
  32. 2023Decomposed Prompting for Machine Translation Between Related Languages using Large Language Modelscited by 3
  33. 2022Bootstrapping Multilingual Semantic Parsers using Large Language Modelscited by 2
  34. 2023Evaluation of medium-large Language Models at zero-shot closed book generative question answeringcited by 2
  35. 2023Low-Resource Compositional Semantic Parsing with Concept Pretrainingcited by 1
  36. 2023Should ChatGPT and Bard Share Revenue with Their Data Providers? A New Business Model for the AI Eracited by 1
  37. 2023Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Modelscited by 1
  38. 2023Alquist 5.0: Dialogue Trees Meet Generative Models. A Novel Approach for Enhancing SocialBot Conversationscited by 1

50 most cited of 57 All 57 (JSON)