mlsanityData to 14 Oct 2026

Scaling Instruction-Finetuned Language Models

arXiv 2210.11416 ↗First posted 20 Oct 2022cs.LG cs.CLcited by 3,024

Not on the map. Why?

Cited by 347 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints, which achieve strong few-shot performance even compared to much larger models, such as PaLM 62B. Overall, instruction finetuning is a general method for improving the performance and usability of pretrained language models.

Key facts. Cited by 3,024 papers in the map (data through 14 Oct 2026). Builds on 74 of its 106 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

74 of 106 references.

References · 74

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
38 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  13. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  14. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  15. 2018Model Cards for Model Reportingcited by 1,080
  16. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  17. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  18. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  19. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  20. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  21. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  22. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  23. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  24. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  25. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  26. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  27. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  28. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  29. 2021MetaICL: Learning to Learn In Contextcited by 454
  30. 2018Gender Bias in Coreference Resolutioncited by 442
  31. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 404
  32. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  33. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 337
  34. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  35. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  36. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  37. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  38. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 171

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 39; 2023: 1050; 2024: 1085; 2025: 572; 2026: 278.

3,024 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
38 more
  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  9. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  13. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  14. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  15. 2023PaLM 2 Technical Reportcited by 1,162
  16. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  17. 2023LIMA: Less Is More for Alignmentcited by 1,100
  18. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  19. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  20. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  21. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  22. 2023A Survey on Multimodal Large Language Modelscited by 909
  23. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  24. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  25. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  26. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  27. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  28. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  29. 2024Chronos: Learning the Language of Time Seriescited by 759
  30. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  31. 2022Galactica: A Large Language Model for Sciencecited by 744
  32. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  33. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  34. 2024Yi: Open Foundation Models by 01.AIcited by 719
  35. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  36. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  37. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  38. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698

50 most cited of 3,024 All 3,024 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
  2. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  3. 2024Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant
  4. 2026FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
  5. 2026Do Instruction-Tuned Models Always Perform Better Than Base Models? Evidence from Math and Domain-Shifted Benchmarks
  6. 2025Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact
  7. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
  8. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning