mlsanityData to 7 Oct 2026

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization

arXiv 2212.12017 ↗First posted 22 Dec 2022cs.CLcited by 235

Not on the map. Why?

Abstract (arXiv)

Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process. These decisions include the scale and diversity of the instruction-tuning benchmark, different task sampling strategies, fine-tuning with and without demonstrations, training using specialized datasets for reasoning and dialogue, and finally, the fine-tuning objectives themselves. In this paper, we characterize the effect of instruction-tuning decisions on downstream task performance when scaling both model and benchmark sizes. To this end, we create OPT-IML Bench: a large benchmark for Instruction Meta-Learning (IML) of 2000 NLP tasks consolidated into task categories from 8 existing benchmarks, and prepare an evaluation framework to measure three types of model generalizations: to tasks from fully held-out categories, to held-out tasks from seen categories, and to held-out instances from seen tasks. Through the lens of this framework, we first present insights about instruction-tuning decisions as applied to OPT-30B and further exploit these insights to train OPT-IML 30B and 175B, which are instruction-tuned versions of OPT. OPT-IML demonstrates all three generalization abilities at both scales on four different evaluation benchmarks with diverse tasks and input formats -- PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG. Not only does it significantly outperform OPT on all benchmarks but is also highly competitive with existing models fine-tuned on each specific benchmark. We release OPT-IML at both scales, together with the OPT-IML Bench evaluation framework.

Key facts. Cited by 235 papers in the map (data through 7 Oct 2026). Builds on 70 of its 85 references that are in the map. First posted on arXiv 22 Dec 2022.

Builds on

70 of 85 references.

References · 70

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
38 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  13. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  14. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  15. 2017Mixed Precision Trainingcited by 1,135
  16. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  17. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  18. 2022Editing Models with Task Arithmeticcited by 1,076
  19. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  20. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  21. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  22. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  23. 2018CoQA: A Conversational Question Answering Challengecited by 674
  24. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  25. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  26. 2020Recipes for building an open-domain chatbotcited by 593
  27. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  28. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  29. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  30. 2021MetaICL: Learning to Learn In Contextcited by 450
  31. 2018Gender Bias in Coreference Resolutioncited by 438
  32. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  33. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  34. 2020The Pushshift Reddit Datasetcited by 289
  35. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  36. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  37. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  38. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 126; 2024: 79; 2025: 24; 2026: 5.

235 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  7. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  8. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  9. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  10. 2023A Survey on Multimodal Large Language Modelscited by 898
  11. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  12. 2022A Survey on In-context Learningcited by 717
38 more
  1. 2023Instruction Tuning with GPT-4cited by 676
  2. 2024A Survey on Large Language Models for Code Generationcited by 646
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  6. 2024Large Language Models: A Surveycited by 506
  7. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  8. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  9. 2023Augmented Language Models: a Surveycited by 398
  10. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  11. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  12. 2023Tool Learning with Foundation Modelscited by 302
  13. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  14. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  15. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  16. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  17. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  18. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  19. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  20. 2024Generative Representational Instruction Tuningcited by 227
  21. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  22. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  23. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  24. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  25. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  26. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  27. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  28. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  29. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 174
  30. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  31. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  32. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  33. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  34. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  35. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  36. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 128
  37. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  38. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122

50 most cited of 235 All 235 (JSON)