mlsanityData to 7 Oct 2026

Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator

arXiv 2206.08082 ↗First posted 16 Jun 2022cs.CLcited by 68

On the map: Synthetic data and self-improvement for LLMs → Self-training for few-shot NLP

Abstract (arXiv)

Large-scale pre-trained language models (PLMs) are well-known for being capable of solving a task simply by conditioning a few input-label pairs dubbed demonstrations on a prompt without being explicitly tuned for the desired downstream task. Such a process (i.e., in-context learning), however, naturally leads to high reliance on the demonstrations which are usually selected from external datasets. In this paper, we propose self-generated in-context learning (SG-ICL), which generates demonstrations for in-context learning from PLM itself to minimize the reliance on the external demonstration. We conduct experiments on four different text classification tasks and show SG-ICL significantly outperforms zero-shot learning and is generally worth approximately 0.6 gold training samples. Moreover, our generated demonstrations show more consistent performance with low variance compared to randomly selected demonstrations from the training dataset.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 13 of its 17 references that are in the map. First posted on arXiv 16 Jun 2022.

Builds on

13 of 17 references.

References · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  8. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  10. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  11. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  12. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
1 more
  1. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 21; 2024: 26; 2025: 14; 2026: 6.

68 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2022A Survey on In-context Learningcited by 717
  3. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  4. 2024Many-Shot In-Context Learningcited by 203
  5. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 188
  6. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  7. 2023Large Language Models as Analogical Reasonerscited by 107
  8. 2023Adversarial Demonstration Attacks on Large Language Modelscited by 58
  9. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
  10. 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
  11. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
  12. 2023Large Language Models can be Guided to Evade AI-Generated Text Detectioncited by 43
38 more
  1. 2023Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrationscited by 42
  2. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  3. 2025Emergent Abilities in Large Language Models: A Surveycited by 34
  4. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 31
  5. 2023Instruct Me More! Random Prompting for Visual In-Context Learningcited by 30
  6. 2024PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learningcited by 28
  7. 2024Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimizationcited by 28
  8. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 27
  9. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 26
  10. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 25
  11. 2023Auto-ICL: In-Context Learning without Human Supervisioncited by 18
  12. 2024From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systemscited by 17
  13. 2023Improving Probability-based Prompt Selection Through Unified Evaluation and Analysiscited by 13
  14. 2023More Samples or More Prompts? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineeringcited by 13
  15. 2025Where to show Demos in Your Prompt: A Positional Bias of In-Context Learningcited by 13
  16. 2025Revisiting Chain-of-Thought Prompting: Zero-shot Can Be Stronger than Few-shotcited by 12
  17. 2023Are Human-generated Demonstrations Necessary for In-context Learning?cited by 11
  18. 2024Demonstration Augmentation for Zero-shot In-context Learningcited by 11
  19. 2024The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotatorscited by 10
  20. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  21. 2024Token-based Decision Criteria Are Suboptimal in In-context Learningcited by 9
  22. 2024Self-Evolving GPT: A Lifelong Autonomous Experiential Learnercited by 9
  23. 2024Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?cited by 8
  24. 2024FastGAS: Fast Graph-based Annotation Selection for In-Context Learningcited by 8
  25. 2023On the Opportunities of Green Computing: A Surveycited by 7
  26. 2024Chain of Evidences and Evidence to Generate: Prompting for Context Grounded and Retrieval Augmented Reasoningcited by 7
  27. 2023Dynamic Demonstrations Controller for In-Context Learningcited by 6
  28. 2025DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfercited by 5
  29. 2023KwaiYiiMath: Technical Reportcited by 4
  30. 2024LLM can Achieve Self-Regulation via Hyperparameter Aware Generationcited by 4
  31. 2024Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Modelscited by 4
  32. 2024Dynamic Few-Shot Learning for Knowledge Graph Question Answeringcited by 4
  33. 2026Experience Transfer for Multimodal LLM Agents in Minecraft Gamecited by 4
  34. 2024Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contextscited by 3
  35. 2024In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Taskscited by 3
  36. 2023PACIT: Unlocking the Power of Examples for Better In-Context Instruction Tuningcited by 2
  37. 2024Effective In-Context Example Selection through Data Compressioncited by 2
  38. 2024Inductive Linguistic Reasoning with Large Language Modelscited by 2

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  2. 2024In-Context Language Learning: Architectures and Algorithms
  3. 2023Dr.ICL: Demonstration-Retrieved In-context Learning
  4. 2025Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines
  5. 2025On the Relationship Between the Choice of Representation and In-Context Learning
  6. 2025Leveraging In-Context Learning for Language Model Agents
  7. 2024Concept-aware Data Construction Improves In-context Learning of Language Models
  8. 2025Refract ICL: Rethinking Example Selection in the Era of Million-Token Models