# Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations

arXiv: https://arxiv.org/abs/2212.09865 · HTML: https://mlsanity.com/paper/2212.09865/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 19 Dec 2022.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://mlsanity.com/paper/2108.04106/) (2021)
- [Impact of Pretraining Term Frequencies on Few-Shot Reasoning](https://mlsanity.com/paper/2202.07206/) (2022)
- [DEMix Layers: Disentangling Domains for Modular Language Modeling](https://mlsanity.com/paper/2108.05036/) (2021)
- [Semantic-Oriented Unlabeled Priming for Large-Scale Language Models](https://mlsanity.com/paper/2202.06133/) (2022)

## Built on by (most cited first, 30 of 70)

- [The Learnability of In-Context Learning](https://mlsanity.com/paper/2303.07895/) (2023)
- [Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements](https://mlsanity.com/paper/2401.06766/) (2024)
- [Reliable, Adaptable, and Attributable Language Models with Retrieval](https://mlsanity.com/paper/2403.03187/) (2024)
- [A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners](https://mlsanity.com/paper/2406.11050/) (2024)
- [Dr.ICL: Demonstration-Retrieved In-context Learning](https://mlsanity.com/paper/2305.14128/) (2023)
- [In-context Learning with Retrieved Demonstrations for Language Models: A Survey](https://mlsanity.com/paper/2401.11624/) (2024)
- [Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models](https://mlsanity.com/paper/2302.00618/) (2023)
- [Dual Operating Modes of In-Context Learning](https://mlsanity.com/paper/2402.18819/) (2024)
- [Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations](https://mlsanity.com/paper/2305.15035/) (2023)
- [A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoning](https://mlsanity.com/paper/2501.02497/) (2025)
- [MoT: Memory-of-Thought Enables ChatGPT to Self-Improve](https://mlsanity.com/paper/2305.05181/) (2023)
- [Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following](https://mlsanity.com/paper/2302.14691/) (2023)
- [How to Configure Good In-Context Sequence for Visual Question Answering](https://mlsanity.com/paper/2312.01571/) (2023)
- [LLMs Are In-Context Bandit Reinforcement Learners](https://mlsanity.com/paper/2410.05362/) (2024)
- [Large Language Models are Strong Zero-Shot Retriever](https://mlsanity.com/paper/2304.14233/) (2023)
- [C-ICL: Contrastive In-context Learning for Information Extraction](https://mlsanity.com/paper/2402.11254/) (2024)
- [TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration](https://mlsanity.com/paper/2505.17098/) (2025)
- [IDAS: Intent Discovery with Abstractive Summarization](https://mlsanity.com/paper/2305.19783/) (2023)
- [Supervised Knowledge Makes Large Language Models Better In-context Learners](https://mlsanity.com/paper/2312.15918/) (2023)
- [Dissecting Chain-of-Thought: Compositionality through In-Context Filtering and Learning](https://mlsanity.com/paper/2305.18869/) (2023)
- [Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruning](https://mlsanity.com/paper/2310.12774/) (2023)
- [On the Noise Robustness of In-Context Learning for Text Generation](https://mlsanity.com/paper/2405.17264/) (2024)
- [Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classifications](https://mlsanity.com/paper/2310.14607/) (2023)
- [Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models](https://mlsanity.com/paper/2311.08921/) (2023)
- [Exploring Automated Distractor Generation for Math Multiple-choice Questions via Large Language Models](https://mlsanity.com/paper/2404.02124/) (2024)
- [Universal Self-Adaptive Prompting](https://mlsanity.com/paper/2305.14926/) (2023)
- [Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition](https://mlsanity.com/paper/2410.05603/) (2024)
- [Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis](https://mlsanity.com/paper/2305.14877/) (2023)
- [The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis](https://mlsanity.com/paper/2402.12976/) (2024)
- [Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Tasks](https://mlsanity.com/paper/2405.10548/) (2024)

## Abstract (arXiv)

Although large language models can be prompted for both zero- and few-shot learning, performance drops significantly when no demonstrations are available. In this paper, we introduce Z-ICL, a new zero-shot method that closes the gap by constructing pseudo-demonstrations for a given test input using a raw text corpus. Concretely, pseudo-demonstrations are constructed by (1) finding the nearest neighbors to the test input from the corpus and pairing them with random task labels, and (2) applying a set of techniques to reduce the amount of direct copying the model does from the resulting demonstrations. Evaluation on nine classification datasets shows that Z-ICL outperforms previous zero-shot methods by a significant margin, and is on par with in-context learning with labeled training data in the few-shot setting. Overall, Z-ICL provides a significantly higher estimate of the zero-shot performance levels of a model, and supports future efforts to develop better pseudo-demonstrations that further improve zero-shot results.