mlsanityData to 7 Oct 2026

MetaICL: Learning to Learn In Context

arXiv 2110.15943 ↗First posted 29 Oct 2021cs.CL cs.AIcited by 450

Not on the map. Why?

Cited by 29 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce MetaICL (Meta-training for In-Context Learning), a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learning on a large set of training tasks. This meta-training enables the model to more effectively learn a new task in context at test time, by simply conditioning on a few training examples with no parameter updates or task-specific templates. We experiment on a large, diverse collection of tasks consisting of 142 NLP datasets including classification, question answering, natural language inference, paraphrase detection and more, across seven different meta-training/target splits. MetaICL outperforms a range of baselines including in-context learning without meta-training and multi-task learning followed by zero-shot transfer. We find that the gains are particularly significant for target tasks that have domain shifts from the meta-training tasks, and that using a diverse set of the meta-training tasks is key to improvements. We also show that MetaICL approaches (and sometimes beats) the performance of models fully finetuned on the target task, and outperforms much bigger models with nearly 8x parameters. Finally, we show that MetaICL is complementary to human-written instructions, and the best performance can be achieved by combining both approaches.

Key facts. Cited by 450 papers in the map (data through 7 Oct 2026). Builds on 83 of its 139 references that are in the map. First posted on arXiv 29 Oct 2021.

Builds on

83 of 139 references.

References · 83

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
38 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  2. 2020Extracting Training Data from Large Language Modelscited by 2,144
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2019Language Models as Knowledge Bases?cited by 1,922
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  10. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2017Mixed Precision Trainingcited by 1,135
  13. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  14. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  15. 2018Neural Network Acceptability Judgmentscited by 952
  16. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  17. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  18. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  19. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  20. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  21. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  22. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  23. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  24. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  25. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  26. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  27. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  28. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  29. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  30. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  31. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 403
  32. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  33. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  34. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  35. 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 361
  36. 2021True Few-Shot Learning with Language Modelscited by 343
  37. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  38. 2016NewsQA: A Machine Comprehension Datasetcited by 330

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 61; 2023: 153; 2024: 123; 2025: 74; 2026: 39.

450 papers.

Sort:

By area:

  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  7. 2022Editing Models with Task Arithmeticcited by 1,076
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  9. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  10. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  12. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
38 more
  1. 2022A Survey on In-context Learningcited by 717
  2. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  3. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  4. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  5. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  6. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  7. 2023Scaling Data-Constrained Language Modelscited by 391
  8. 2023Larger language models do in-context learning differentlycited by 386
  9. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  10. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  11. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  12. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  13. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  14. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  15. 2023Poisoning Language Models During Instruction Tuningcited by 254
  16. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  17. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  18. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  19. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  20. 2024Generative Representational Instruction Tuningcited by 227
  21. 2023Improving CLIP Training with Language Rewritescited by 216
  22. 2023Efficient Large Language Models: A Surveycited by 183
  23. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  24. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  25. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  26. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  27. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  28. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  29. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  30. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 153
  31. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  32. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  33. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  34. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  35. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  36. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  37. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  38. 2022Learning by Distilling Contextcited by 128

50 most cited of 450 All 450 (JSON)