mlsanityData to 7 Oct 2026

Improving In-Context Few-Shot Learning via Self-Supervised Training

arXiv 2205.01703 ↗First posted 3 May 2022cs.CLcited by 56

Not on the map. Why?

Abstract (arXiv)

Self-supervised pretraining has made few-shot learning possible for many NLP tasks. But the pretraining objectives are not typically adapted specifically for in-context few-shot learning. In this paper, we propose to use self-supervision in an intermediate training stage between pretraining and downstream few-shot usage with the goal to teach the model to perform in-context few shot learning. We propose and evaluate four self-supervised objectives on two benchmarks. We find that the intermediate self-supervision stage produces models that outperform strong baselines. Ablation study shows that several factors affect the downstream performance, such as the amount of training data and the diversity of the self-supervised objectives. Human-annotated cross-task supervision and self-supervision are complementary. Qualitative analysis suggests that the self-supervised-trained models are better at following task requirements.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 67 of its 98 references that are in the map. First posted on arXiv 3 May 2022.

Builds on

67 of 98 references.

References · 67

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
38 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  6. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  8. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  10. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  11. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  12. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  13. 2021MetaICL: Learning to Learn In Contextcited by 450
  14. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  15. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  16. 2019Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachcited by 299
  17. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  18. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248
  19. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  20. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  21. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  22. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  23. 2021How Many Data Points is a Prompt Worth?cited by 197
  24. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  25. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  26. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  27. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  28. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  29. 2021Entailment as Few-Shot Learnercited by 122
  30. 2019"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understandingcited by 121
  31. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  32. 2019Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoningcited by 100
  33. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  34. 2020Self-training Improves Pre-training for Natural Language Understandingcited by 96
  35. 2020Few-Shot Text Generation with Pattern-Exploiting Trainingcited by 87
  36. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  37. 2020Learning from Task Descriptionscited by 78
  38. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 22; 2024: 19; 2025: 7; 2026: 4.

56 papers.

Sort:

By area:

  1. 2022A Survey on In-context Learningcited by 717
  2. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  3. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  4. 2023Revisiting Relation Extraction in the era of Large Language Modelscited by 100
  5. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  6. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  7. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  8. 2023ChatHaruhi: Reviving Anime Character in Reality via Large Language Modelcited by 57
  9. 2024Comp4D: LLM-Guided Compositional 4D Scene Generationcited by 50
  10. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learningcited by 47
  11. 2024ARC: A Generalist Graph Anomaly Detector with In-Context Learningcited by 46
  12. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
38 more
  1. 2023Small Models are Valuable Plug-ins for Large Language Modelscited by 42
  2. 2023Using In-Context Learning to Improve Dialogue Safetycited by 41
  3. 2023Pre-Training to Learn in Contextcited by 41
  4. 2023How are Prompts Different in Terms of Sensitivity?cited by 34
  5. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  6. 2024Meta-Prompting for Automating Zero-shot Visual Recognition with LLMscited by 28
  7. 2023Interleaving Pre-Trained Language Models and Large Language Models for Zero-Shot NL2SQL Generationcited by 24
  8. 2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
  9. 2022Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalizationcited by 15
  10. 2024Investigating Multi-Hop Factual Shortcuts in Knowledge Editing of Large Language Modelscited by 14
  11. 2024Retrieval-Augmented Data Augmentation for Low-Resource Domain Taskscited by 12
  12. 2023Aladdin: Zero-Shot Hallucination of Stylized 3D Assets from Abstract Scene Descriptionscited by 11
  13. 2024Demonstration Augmentation for Zero-shot In-context Learningcited by 11
  14. 2024"In Dialogues We Learn": Towards Personalized Dialogue Without Pre-defined Profiles through In-Dialogue Learningcited by 10
  15. 2024Token-based Decision Criteria Are Suboptimal in In-context Learningcited by 9
  16. 2023Scaling In-Context Demonstrations with Structured Attentioncited by 8
  17. 2023Learning In-context Learning for Named Entity Recognitioncited by 7
  18. 2023On the Opportunities of Green Computing: A Surveycited by 7
  19. 2023MMICT: Boosting Multi-Modal Fine-Tuning with In-Context Examplescited by 7
  20. 2022Few-shot Adaptation Works with UnpredicTable Datacited by 6
  21. 2023Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learningcited by 6
  22. 2023Surpassing GPT-4 Medical Coding with a Two-Stage Approachcited by 6
  23. 2023Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Modelscited by 6
  24. 2023Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Taskscited by 5
  25. 2023Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learningcited by 5
  26. 2024Characteristic AI Agents via Large Language Modelscited by 5
  27. 2024Enhancing Text Annotation through Rationale-Driven Collaborative Few-Shot Promptingcited by 5
  28. 2024PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Bankscited by 5
  29. 2024Unlocking LLMs: Addressing Scarce Data and Bias Challenges in Mental Healthcited by 5
  30. 2025ImpRAG: Retrieval-Augmented Generation with Implicit Queriescited by 5
  31. 2023SINC: Self-Supervised In-Context Learning for Vision-Language Taskscited by 4
  32. 2025Understanding Before Reasoning: Enhancing Chain-of-Thought with Iterative Summarization Pre-Promptingcited by 4
  33. 2026Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Taskscited by 4
  34. 2025You Only Fine-tune Once: Many-Shot In-Context Fine-Tuning for Large Language Modelscited by 3
  35. 2024AICL: Action In-Context Learning for Video Diffusion Modelcited by 2
  36. 2025Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regressioncited by 2
  37. 2026FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimizationcited by 2
  38. 2025More Bias, Less Bias: BiasPrompting for Enhanced Multiple-Choice Question Answeringcited by 1

50 most cited of 56 All 56 (JSON)