# Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations

arXiv: https://arxiv.org/abs/2305.15035 · HTML: https://mlsanity.com/paper/2305.15035/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 36 of its 43 references that are in the map.
- First posted on arXiv 24 May 2023.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-training for few-shot NLP](https://mlsanity.com/synth/self-training-for-few-shot-nlp/)

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [Complexity-Based Prompting for Multi-Step Reasoning](https://mlsanity.com/paper/2210.00720/) (2022)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [Data Distributional Properties Drive Emergent In-Context Learning in Transformers](https://mlsanity.com/paper/2205.05055/) (2022)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [Demystifying Prompts in Language Models via Perplexity Estimation](https://mlsanity.com/paper/2212.04037/) (2022)
- [Active Example Selection for In-Context Learning](https://mlsanity.com/paper/2211.04486/) (2022)
- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://mlsanity.com/paper/2108.04106/) (2021)

## Built on by (most cited first, 30 of 42)

- [InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales](https://mlsanity.com/paper/2406.13629/) (2024)
- [In-Context Learning with Iterative Demonstration Selection](https://mlsanity.com/paper/2310.09881/) (2023)
- [Many-Shot In-Context Learning in Multimodal Foundation Models](https://mlsanity.com/paper/2405.09798/) (2024)
- [StreamBench: Towards Benchmarking Continuous Improvement of Language Agents](https://mlsanity.com/paper/2406.08747/) (2024)
- [A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoning](https://mlsanity.com/paper/2501.02497/) (2025)
- [SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning](https://mlsanity.com/paper/2407.12874/) (2024)
- [Distill Visual Chart Reasoning Ability from LLMs to MLLMs](https://mlsanity.com/paper/2410.18798/) (2024)
- [Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments](https://mlsanity.com/paper/2406.11370/) (2024)
- [From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generation](https://mlsanity.com/paper/2502.00330/) (2025)
- [Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruning](https://mlsanity.com/paper/2310.12774/) (2023)
- [How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks](https://mlsanity.com/paper/2507.01955/) (2025)
- [Demonstration Augmentation for Zero-shot In-context Learning](https://mlsanity.com/paper/2406.01224/) (2024)
- [Self-Evolving GPT: A Lifelong Autonomous Experiential Learner](https://mlsanity.com/paper/2407.08937/) (2024)
- [Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?](https://mlsanity.com/paper/2404.12728/) (2024)
- [Scaling Laws for Many-Shot In-Context Learning with Self-Generated Annotations](https://mlsanity.com/paper/2503.03062/) (2025)
- [What Makes a Good Natural Language Prompt?](https://mlsanity.com/paper/2506.06950/) (2025)
- [Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning](https://mlsanity.com/paper/2309.14771/) (2023)
- [DAWN-ICL: Strategic Planning of Problem-solving Trajectories for Zero-Shot In-Context Learning](https://mlsanity.com/paper/2410.20215/) (2024)
- [ICXML: An In-Context Learning Framework for Zero-Shot Extreme Multi-Label Classification](https://mlsanity.com/paper/2311.09649/) (2023)
- [Large Language Models are In-context Teachers for Knowledge Reasoning](https://mlsanity.com/paper/2311.06985/) (2023)
- [Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Models](https://mlsanity.com/paper/2404.00884/) (2024)
- [Let's Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Models](https://mlsanity.com/paper/2410.21728/) (2024)
- [Tuning-Free Personalized Alignment via Trial-Error-Explain In-Context Learning](https://mlsanity.com/paper/2502.08972/) (2025)
- [TestNUC: Enhancing Test-Time Computing Approaches and Scaling through Neighboring Unlabeled Data Consistency](https://mlsanity.com/paper/2502.19163/) (2025)
- [Exploring Large Language Models for Product Attribute Value Identification](https://mlsanity.com/paper/2409.12695/) (2024)
- [In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks](https://mlsanity.com/paper/2410.01548/) (2024)
- [Document-Level Event Extraction with Definition-Driven ICL](https://mlsanity.com/paper/2408.05566/) (2024)
- [PICLe: Pseudo-Annotations for In-Context Learning in Low-Resource Named Entity Detection](https://mlsanity.com/paper/2412.11923/) (2024)
- [Exploring In-context Example Generation for Machine Translation](https://mlsanity.com/paper/2506.00507/) (2025)
- [Online In-Context Distillation for Low-Resource Vision Language Models](https://mlsanity.com/paper/2510.18117/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have exhibited striking in-context learning (ICL) ability to adapt to target tasks with a few input-output demonstrations. For better ICL, different methods are proposed to select representative demonstrations from existing training corpora. However, such settings are not aligned with real-world practices, as end-users usually query LMs without access to demonstration pools. In this work, we introduce Self-ICL -- a simple framework which bootstraps LMs' intrinsic capabilities to perform zero-shot ICL. Given a test input, Self-ICL first prompts the model to generate pseudo-inputs. Next, the model predicts pseudo-labels for the pseudo-inputs via zero-shot prompting. Finally, we perform ICL for the test input with the pseudo-input-label pairs as demonstrations. Evaluation on 23 BIG-Bench Hard tasks shows Self-ICL outperforms zero-shot baselines on both average accuracy and head-to-head comparison. Moreover, with zero-shot chain-of-thought, Self-ICL achieves results comparable to using real demonstrations. Additionally, we conduct a range of analyses to validate Self-ICL's effectiveness and provide insights for its behaviors under different settings.