Data to 14 Oct 2026

Meta-learning via Language Model In-context Tuning

arXiv 2110.07814 ↗First posted 15 Oct 2021cs.CL cs.LGcited by 164

Not on the map. Why?

Abstract (arXiv)

The goal of meta-learning is to learn to adapt to a new task with only a few labeled examples. To tackle this problem in NLP, we propose textitin-context tuning, which recasts adaptation and prediction as a simple sequence prediction problem: to form the input sequence, we concatenate the task instruction, the labeled examples, and the target input to predict; to meta-train the model to learn from in-context examples, we fine-tune a pre-trained language model (LM) to predict the target label from the input sequences on a collection of tasks. We benchmark our method on two collections of text classification tasks: LAMA and BinaryClfs. Compared to first-order MAML which adapts the model with gradient descent, our method better leverages the inductive bias of LMs to perform pattern matching, and outperforms MAML by an absolute 6% AUC ROC score on BinaryClfs, with increasing advantage w.r.t. model size. Compared to non-fine-tuned in-context learning (i.e. prompting a raw LM), in-context tuning directly learns to learn from in-context examples. On BinaryClfs, in-context tuning improves the average AUC-ROC score by an absolute 10%, and reduces the variance with respect to example ordering by 6x and example choices by 2x.

Key facts. Cited by 164 papers in the map (data through 14 Oct 2026). Builds on 48 of its 64 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

48 of 64 references.

References · 48

2015Deep Residual Learning for Image Recognitioncited by 52,096
2020Language Models are Few-Shot Learnerscited by 32,419
2020Scaling Laws for Neural Language Modelscited by 6,586
2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
2019Language Models as Knowledge Bases?cited by 1,936
2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
2019Generating Long Sequences with Sparse Transformerscited by 1,546

36 more

2020Linformer: Self-Attention with Linear Complexitycited by 1,474
2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
2018On First-Order Meta-Learning Algorithmscited by 1,018
2019How Can We Know What Language Models Know?cited by 959
2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
2021Robust fine-tuning of zero-shot modelscited by 797
2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
2019Meta-Learning with Implicit Gradientscited by 525
2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
2020Towards a Human-like Open-Domain Chatbotcited by 470
2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
2018Efficient Attention: Attention with Linear Complexitiescited by 368
2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
2021True Few-Shot Learning with Language Modelscited by 344
2018How to train your MAMLcited by 309
2019Massively Multilingual Neural Machine Translationcited by 246
2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
2018Meta-Learning for Low-Resource Neural Machine Translationcited by 105
2019E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTcited by 96
2016Low Data Drug Discovery with One-shot Learningcited by 91
2019Induction Networks for Few-Shot Text Classificationcited by 64
2018Diverse Few-Shot Text Classification with Multiple Metricscited by 61
2019Learning to Few-Shot Learn Across Diverse Natural Language Classification Taskscited by 57
2019Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detectioncited by 54
2019Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Taskscited by 48
2021Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and Systemcited by 30
2019From Zero-Shot Learning to Cold-Start Recommendationcited by 29
2020Learning to Learn to Disambiguate: Meta-Learning for Few-Shot Word Sense Disambiguationcited by 13
2020Fine-Grained Analysis of Cross-Linguistic Syntactic Divergencescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 28; 2023: 45; 2024: 43; 2025: 33; 2026: 14.

164 papers.

Sort:

By area:

2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
2022A Survey on In-context Learningcited by 721
2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
2021MetaICL: Learning to Learn In Contextcited by 454
2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
2024Large Language Model-Brained GUI Agents: A Surveycited by 162

38 more

2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 139
2022Learning by Distilling Contextcited by 138
2023Dolphins: Multimodal Language Model for Drivingcited by 124
2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 109
2022Successive Prompting for Decomposing Complex Questionscited by 94
2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 81
2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
2023Boosting Language Models Reasoning with Chain-of-Knowledge Promptingcited by 77
2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 72
2022Complementary Explanations for Effective In-Context Learningcited by 71
2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
2024Instructional Fingerprinting of Large Language Modelscited by 68
2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
2023Dynamic Planning with a LLMcited by 61
2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
2023FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Modelscited by 51
2023kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inferencecited by 47
2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 46
2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
2023Pre-Training to Learn in Contextcited by 42
2023Forward-Backward Reasoning in Large Language Models for Mathematical Verificationcited by 40
2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 40
2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 37
2023Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selectioncited by 37
2024LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformerscited by 37
2024In-Context Fine-Tuning for Time-Series Foundation Modelscited by 35
2024In-Context Editing: Learning Knowledge from Self-Induced Distributionscited by 34
2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 33
2022Weakly Supervised Data Augmentation Through Prompting for Dialogue Understandingcited by 32
2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
2022Task Ambiguity in Humans and Language Modelscited by 28
2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 27

50 most cited of 164 All 164 (JSON)