# Prompt Consistency for Zero-Shot Task Generalization

arXiv: https://arxiv.org/abs/2205.00049 · HTML: https://mlsanity.com/paper/2205.00049/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 57 references that are in the map.
- First posted on arXiv 29 Apr 2022.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)

## Built on by (most cited first, 30 of 67)

- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [LM vs LM: Detecting Factual Errors via Cross Examination](https://mlsanity.com/paper/2305.13281/) (2023)
- [Understanding Zero-Shot Adversarial Robustness for Large-Scale Models](https://mlsanity.com/paper/2212.07016/) (2022)
- [On the Relation between Sensitivity and Accuracy in In-context Learning](https://mlsanity.com/paper/2209.07661/) (2022)
- [The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems](https://mlsanity.com/paper/2503.03750/) (2025)
- [Language Models in the Loop: Incorporating Prompting into Weak Supervision](https://mlsanity.com/paper/2205.02318/) (2022)
- [Improving Diffusion Models for Scene Text Editing with Dual Encoders](https://mlsanity.com/paper/2304.05568/) (2023)
- [Benchmarking and Improving Generator-Validator Consistency of Language Models](https://mlsanity.com/paper/2310.01846/) (2023)
- [Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justifications](https://mlsanity.com/paper/2403.16073/) (2024)
- [Measuring Reliability of Large Language Models through Semantic Consistency](https://mlsanity.com/paper/2211.05853/) (2022)
- [On the Worst Prompt Performance of Large Language Models](https://mlsanity.com/paper/2406.10248/) (2024)
- [Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer](https://mlsanity.com/paper/2309.07929/) (2023)
- [Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs](https://mlsanity.com/paper/2305.14279/) (2023)
- [PURPLE: Making a Large Language Model a Better SQL Writer](https://mlsanity.com/paper/2403.20014/) (2024)
- [Doubly Right Object Recognition: A Why Prompt for Visual Rationales](https://mlsanity.com/paper/2212.06202/) (2022)
- [Predicting Question-Answering Performance of Large Language Models through Semantic Consistency](https://mlsanity.com/paper/2311.01152/) (2023)
- [Zero-Shot Text Classification with Self-Training](https://mlsanity.com/paper/2210.17541/) (2022)
- [reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs](https://mlsanity.com/paper/2503.11751/) (2025)
- [A Survey on the Honesty of Large Language Models](https://mlsanity.com/paper/2409.18786/) (2024)
- [Knowledge-Augmented Language Model Verification](https://mlsanity.com/paper/2310.12836/) (2023)
- [Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification](https://mlsanity.com/paper/2307.01972/) (2023)
- [On the Consistency of Video Large Language Models in Temporal Comprehension](https://mlsanity.com/paper/2411.12951/) (2024)
- [Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization](https://mlsanity.com/paper/2210.09175/) (2022)
- [Semantic Consistency for Assuring Reliability of Large Language Models](https://mlsanity.com/paper/2308.09138/) (2023)
- [A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness](https://mlsanity.com/paper/2312.01082/) (2023)
- [Enhancing Semantic Consistency of Large Language Models through Model Editing: An Interpretability-Oriented Approach](https://mlsanity.com/paper/2501.11041/) (2025)
- [Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models](https://mlsanity.com/paper/2210.14803/) (2022)

## Abstract (arXiv)

One of the most impressive results of recent NLP history is the ability of pre-trained language models to solve new tasks in a zero-shot setting. To achieve this, NLP tasks are framed as natural language prompts, generating a response indicating the predicted output. Nonetheless, the performance in such settings often lags far behind its supervised counterpart, suggesting a large space for potential improvement. In this paper, we explore methods to utilize unlabeled data to improve zero-shot performance. Specifically, we take advantage of the fact that multiple prompts can be used to specify a single task, and propose to regularize prompt consistency, encouraging consistent predictions over this diverse set of prompts. Our method makes it possible to fine-tune the model either with extra unlabeled training data, or directly on test input at inference time in an unsupervised manner. In experiments, our approach outperforms the state-of-the-art zero-shot learner, T0 (Sanh et al., 2022), on 9 out of 11 datasets across 4 NLP tasks by up to 10.6 absolute points in terms of accuracy. The gains are often attained with a small number of unlabeled examples.