# Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator

arXiv: https://arxiv.org/abs/2206.08082 · HTML: https://mlsanity.com/paper/2206.08082/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 13 of its 17 references that are in the map.
- First posted on arXiv 16 Jun 2022.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-training for few-shot NLP](https://mlsanity.com/synth/self-training-for-few-shot-nlp/)

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Unsupervised Commonsense Question Answering with Self-Talk](https://mlsanity.com/paper/2004.05483/) (2020)

## Built on by (most cited first, 30 of 68)

- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [Many-Shot In-Context Learning](https://mlsanity.com/paper/2404.11018/) (2024)
- [Recommender Systems in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2307.02046/) (2023)
- [What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization](https://mlsanity.com/paper/2305.19420/) (2023)
- [Large Language Models as Analogical Reasoners](https://mlsanity.com/paper/2310.01714/) (2023)
- [Adversarial Demonstration Attacks on Large Language Models](https://mlsanity.com/paper/2305.14950/) (2023)
- [Active Learning Principles for In-Context Learning with Large Language Models](https://mlsanity.com/paper/2305.14264/) (2023)
- [Exploring Human-Like Translation Strategy with Large Language Models](https://mlsanity.com/paper/2305.04118/) (2023)
- [Large Language Models can be Guided to Evade AI-Generated Text Detection](https://mlsanity.com/paper/2305.10847/) (2023)
- [Exploring Effective Factors for Improving Visual In-Context Learning](https://mlsanity.com/paper/2304.04748/) (2023)
- [Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations](https://mlsanity.com/paper/2305.15035/) (2023)
- [A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoning](https://mlsanity.com/paper/2501.02497/) (2025)
- [Emergent Abilities in Large Language Models: A Survey](https://mlsanity.com/paper/2503.05788/) (2025)
- [How to Configure Good In-Context Sequence for Visual Question Answering](https://mlsanity.com/paper/2312.01571/) (2023)
- [Instruct Me More! Random Prompting for Visual In-Context Learning](https://mlsanity.com/paper/2311.03648/) (2023)
- [PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learning](https://mlsanity.com/paper/2405.02501/) (2024)
- [Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimization](https://mlsanity.com/paper/2406.15708/) (2024)
- [Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods](https://mlsanity.com/paper/2408.14511/) (2024)
- [IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models](https://mlsanity.com/paper/2310.10873/) (2023)
- [In-Context Learning Enables Robot Action Prediction in LLMs](https://mlsanity.com/paper/2410.12782/) (2024)
- [Auto-ICL: In-Context Learning without Human Supervision](https://mlsanity.com/paper/2311.09263/) (2023)
- [From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems](https://mlsanity.com/paper/2405.19883/) (2024)
- [Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis](https://mlsanity.com/paper/2305.14877/) (2023)
- [More Samples or More Prompts? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineering](https://mlsanity.com/paper/2311.09782/) (2023)
- [Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning](https://mlsanity.com/paper/2507.22887/) (2025)
- [Revisiting Chain-of-Thought Prompting: Zero-shot Can Be Stronger than Few-shot](https://mlsanity.com/paper/2506.14641/) (2025)
- [Are Human-generated Demonstrations Necessary for In-context Learning?](https://mlsanity.com/paper/2309.14681/) (2023)
- [Demonstration Augmentation for Zero-shot In-context Learning](https://mlsanity.com/paper/2406.01224/) (2024)

## Abstract (arXiv)

Large-scale pre-trained language models (PLMs) are well-known for being capable of solving a task simply by conditioning a few input-label pairs dubbed demonstrations on a prompt without being explicitly tuned for the desired downstream task. Such a process (i.e., in-context learning), however, naturally leads to high reliance on the demonstrations which are usually selected from external datasets. In this paper, we propose self-generated in-context learning (SG-ICL), which generates demonstrations for in-context learning from PLM itself to minimize the reliance on the external demonstration. We conduct experiments on four different text classification tasks and show SG-ICL significantly outperforms zero-shot learning and is generally worth approximately 0.6 gold training samples. Moreover, our generated demonstrations show more consistent performance with low variance compared to randomly selected demonstrations from the training dataset.