mlsanityData to 7 Oct 2026

Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations

arXiv 2305.15035 ↗First posted 24 May 2023cs.CLcited by 42

On the map: Synthetic data and self-improvement for LLMs → Self-training for few-shot NLP

Abstract (arXiv)

Large language models (LLMs) have exhibited striking in-context learning (ICL) ability to adapt to target tasks with a few input-output demonstrations. For better ICL, different methods are proposed to select representative demonstrations from existing training corpora. However, such settings are not aligned with real-world practices, as end-users usually query LMs without access to demonstration pools. In this work, we introduce Self-ICL -- a simple framework which bootstraps LMs' intrinsic capabilities to perform zero-shot ICL. Given a test input, Self-ICL first prompts the model to generate pseudo-inputs. Next, the model predicts pseudo-labels for the pseudo-inputs via zero-shot prompting. Finally, we perform ICL for the test input with the pseudo-input-label pairs as demonstrations. Evaluation on 23 BIG-Bench Hard tasks shows Self-ICL outperforms zero-shot baselines on both average accuracy and head-to-head comparison. Moreover, with zero-shot chain-of-thought, Self-ICL achieves results comparable to using real demonstrations. Additionally, we conduct a range of analyses to validate Self-ICL's effectiveness and provide insights for its behaviors under different settings.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 36 of its 43 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

36 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator16 Jun

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
23 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  2. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  3. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  5. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  6. 2022Transformers learn in-context by gradient descentcited by 714
  7. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  9. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  10. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  11. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  12. 2023Larger language models do in-context learning differentlycited by 386
  13. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  14. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  15. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  16. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  17. 2022Active Example Selection for In-Context Learningcited by 195
  18. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  19. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  20. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  21. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  22. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 84
  23. 2022Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrationscited by 70

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 17; 2025: 14; 2026: 6.

42 papers.

Sort:

By area:

  1. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  2. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  3. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 64
  4. 2024StreamBench: Towards Benchmarking Continuous Improvement of Language Agentscited by 44
  5. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  6. 2024SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuningcited by 32
  7. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
  8. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgmentscited by 25
  9. 2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generationcited by 20
  10. 2023Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruningcited by 17
  11. 2025How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Taskscited by 14
  12. 2024Demonstration Augmentation for Zero-shot In-context Learningcited by 11
30 more
  1. 2024Self-Evolving GPT: A Lifelong Autonomous Experiential Learnercited by 9
  2. 2024Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?cited by 8
  3. 2025Scaling Laws for Many-Shot In-Context Learning with Self-Generated Annotationscited by 7
  4. 2023Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learningcited by 6
  5. 2024DAWN-ICL: Strategic Planning of Problem-solving Trajectories for Zero-Shot In-Context Learningcited by 6
  6. 2025What Makes a Good Natural Language Prompt?cited by 6
  7. 2023ICXML: An In-Context Learning Framework for Zero-Shot Extreme Multi-Label Classificationcited by 5
  8. 2023Large Language Models are In-context Teachers for Knowledge Reasoningcited by 4
  9. 2024Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Modelscited by 4
  10. 2024Let's Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Modelscited by 4
  11. 2025Tuning-Free Personalized Alignment via Trial-Error-Explain In-Context Learningcited by 4
  12. 2025TestNUC: Enhancing Test-Time Computing Approaches and Scaling through Neighboring Unlabeled Data Consistencycited by 4
  13. 2024Exploring Large Language Models for Product Attribute Value Identificationcited by 3
  14. 2024In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Taskscited by 3
  15. 2024Document-Level Event Extraction with Definition-Driven ICLcited by 2
  16. 2024PICLe: Pseudo-Annotations for In-Context Learning in Low-Resource Named Entity Detectioncited by 2
  17. 2025Exploring In-context Example Generation for Machine Translationcited by 2
  18. 2025Online In-Context Distillation for Low-Resource Vision Language Modelscited by 2
  19. 2026Unlabeled Data Can Provably Enhance In-Context Learning of Transformerscited by 2
  20. 2024DoubleDipper: Improving Long-Context LLMs via Context Recyclingcited by 1
  21. 2025CrossICL: Cross-Task In-Context Learning via Unsupervised Demonstration Transfercited by 1
  22. 2025Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusionscited by 1
  23. 2025Multi-Layer Attention is the Amplifier of Demonstration Effectivenesscited by 1
  24. 2025From Cross-Task Examples to In-Task Prompts: A Graph-Based Pseudo-Labeling Framework for In-context Learningcited by 1
  25. 2026AEMA: Verifiable Evaluation Framework for Trustworthy and Controlled Agentic LLM Systemscited by 1
  26. 2026Self-Improvement as Coherence Optimization: A Theoretical Accountcited by 1
  27. 2026MetaEvo: A Meta-Optimization Framework for Experience-Driven Agent Evolutioncited by 1
  28. 2025ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarkscited by 0
  29. 2026Best-of-Better-N: Generating Pre-Aligned Responses with In-Context Learningcited by 0
  30. 2026Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Refract ICL: Rethinking Example Selection in the Era of Million-Token Models
  2. 2024Many-Shot In-Context Learning
  3. 2023Dr.ICL: Demonstration-Retrieved In-context Learning
  4. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  5. 2023Are Human-generated Demonstrations Necessary for In-context Learning?
  6. 2024In-Context Language Learning: Architectures and Algorithms
  7. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  8. 2025On the Relationship Between the Choice of Representation and In-Context Learning