mlsanityData to 14 Oct 2026

Finetuned Language Models Are Zero-Shot Learners

arXiv 2109.01652 ↗First posted 3 Sep 2021cs.CLcited by 3,780

Not on the map. Why?

Cited by 374 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction tuning -- finetuning language models on a collection of tasks described via instructions -- substantially improves zero-shot performance on unseen tasks. We take a 137B parameter pretrained language model and instruction-tune it on over 60 NLP tasks verbalized via natural language instruction templates. We evaluate this instruction-tuned model, which we call FLAN, on unseen task types. FLAN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 tasks that we evaluate. FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number of finetuning datasets, model scale, and natural language instructions are key to the success of instruction tuning.

Key facts. Cited by 3,780 papers in the map (data through 14 Oct 2026). Builds on 84 of its 170 references that are in the map. First posted on arXiv 3 Sep 2021.

Builds on

84 of 170 references.

References · 84

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  8. 2018Deep contextualized word representationscited by 2,290
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  13. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  14. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  15. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  16. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  17. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  18. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  19. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  20. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  21. 2018Neural Network Acceptability Judgmentscited by 958
  22. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  23. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  24. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  25. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  26. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  27. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  28. 2018CoQA: A Conversational Question Answering Challengecited by 683
  29. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  30. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  31. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  32. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  33. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 426
  34. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  35. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  36. 2018QuAC : Question Answering in Contextcited by 381
  37. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  38. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 33; 2022: 257; 2023: 1049; 2024: 1204; 2025: 767; 2026: 470.

3,780 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  5. 2023A Survey of Large Language Modelscited by 2,576
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  13. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  14. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  15. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  16. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  17. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  18. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  19. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  20. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  21. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  22. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  23. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  24. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  25. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  26. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  27. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  28. 2022Editing Models with Task Arithmeticcited by 1,120
  29. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  30. 2023LIMA: Less Is More for Alignmentcited by 1,100
  31. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  32. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  33. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  34. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  35. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  36. 2022Red Teaming Language Models with Language Modelscited by 1,026
  37. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  38. 2023BloombergGPT: A Large Language Model for Financecited by 939

50 most cited of 3,780 All 3,780 (JSON)