# Auto-ICL: In-Context Learning without Human Supervision

arXiv: https://arxiv.org/abs/2311.09263 · HTML: https://mlsanity.com/paper/2311.09263/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Solving General Arithmetic Word Problems](https://mlsanity.com/paper/1608.01413/) (2016)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://mlsanity.com/paper/2209.10063/) (2022)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [Instruction Induction: From Few Examples to Natural Language Task Descriptions](https://mlsanity.com/paper/2205.10782/) (2022)
- [Large Language Models as Analogical Reasoners](https://mlsanity.com/paper/2310.01714/) (2023)
- [Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator](https://mlsanity.com/paper/2206.08082/) (2022)
- [PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents](https://mlsanity.com/paper/2305.14564/) (2023)
- [Self-Prompting Large Language Models for Zero-Shot Open-Domain QA](https://mlsanity.com/paper/2212.08635/) (2022)

## Built on by (most cited first, 18 of 18)

- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark](https://mlsanity.com/paper/2405.08355/) (2024)
- [Emergent Abilities in Large Language Models: A Survey](https://mlsanity.com/paper/2503.05788/) (2025)
- [Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning](https://mlsanity.com/paper/2507.22887/) (2025)
- [ContextNav: Towards Agentic Multimodal In-Context Learning](https://mlsanity.com/paper/2510.04560/) (2025)
- [Vocabulary-Defined Semantics: Latent Space Clustering for Improving In-Context Learning](https://mlsanity.com/paper/2401.16184/) (2024)
- [Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?](https://mlsanity.com/paper/2404.12728/) (2024)
- [Revisiting In-Context Learning with Long Context Language Models](https://mlsanity.com/paper/2412.16926/) (2024)
- [What Makes a Good Natural Language Prompt?](https://mlsanity.com/paper/2506.06950/) (2025)
- [Let's Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Models](https://mlsanity.com/paper/2410.21728/) (2024)
- [Unlocking In-Context Learning for Natural Datasets Beyond Language Modelling](https://mlsanity.com/paper/2501.06256/) (2025)
- [In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks](https://mlsanity.com/paper/2410.01548/) (2024)
- [Document-Level Event Extraction with Definition-Driven ICL](https://mlsanity.com/paper/2408.05566/) (2024)
- [Inductive Linguistic Reasoning with Large Language Models](https://mlsanity.com/paper/2412.17819/) (2024)
- [Unlabeled Data Can Provably Enhance In-Context Learning of Transformers](https://mlsanity.com/paper/2601.10058/) (2026)
- [Learning to Select Visual In-Context Demonstrations](https://mlsanity.com/paper/2603.26775/) (2026)
- [AEMA: Verifiable Evaluation Framework for Trustworthy and Controlled Agentic LLM Systems](https://mlsanity.com/paper/2601.11903/) (2026)
- [Large Language Models Can Perform Automatic Modulation Classification via Discretized Self-supervised Candidate Retrieval](https://mlsanity.com/paper/2510.00316/) (2025)

## Abstract (arXiv)

With in-context learning ability, the performance of large language models can be significantly boosted when provided with appropriate context. However, existing in-context learning methods mainly rely on human-provided contexts, such as labeled examples and explicit instructions. Writing context by humans is labor-intensive on various tasks and limits the model to tasks manageable by humans. To overcome these limitations, we propose Automatic In-Context Learning framework that enables the model to autonomously generate examples and instructions for problem-solving. With experiments across various models and datasets, results show that model-generated contexts outperform human-annotated contexts, including Few-Shot and Few-Shot-CoT methods, and surpass existing self-generated context methods like Zero-CoT and Auto-CoT.