mlsanityData to 7 Oct 2026

Prompt Consistency for Zero-Shot Task Generalization

arXiv 2205.00049 ↗First posted 29 Apr 2022cs.CL cs.LGcited by 66

Not on the map. Why?

Abstract (arXiv)

One of the most impressive results of recent NLP history is the ability of pre-trained language models to solve new tasks in a zero-shot setting. To achieve this, NLP tasks are framed as natural language prompts, generating a response indicating the predicted output. Nonetheless, the performance in such settings often lags far behind its supervised counterpart, suggesting a large space for potential improvement. In this paper, we explore methods to utilize unlabeled data to improve zero-shot performance. Specifically, we take advantage of the fact that multiple prompts can be used to specify a single task, and propose to regularize prompt consistency, encouraging consistent predictions over this diverse set of prompts. Our method makes it possible to fine-tune the model either with extra unlabeled training data, or directly on test input at inference time in an unsupervised manner. In experiments, our approach outperforms the state-of-the-art zero-shot learner, T0 (Sanh et al., 2022), on 9 out of 11 datasets across 4 NLP tasks by up to 10.6 absolute points in terms of accuracy. The gains are often attained with a small number of unlabeled examples.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 42 of its 57 references that are in the map. First posted on arXiv 29 Apr 2022.

Builds on

42 of 57 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
30 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  7. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  8. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  9. 2022Red Teaming Language Models with Language Modelscited by 983
  10. 2019How Can We Know What Language Models Know?cited by 953
  11. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  12. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  13. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  14. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  15. 2016Sequence-Level Knowledge Distillationcited by 761
  16. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  17. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  18. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  19. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  20. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  21. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  22. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  23. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  24. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  25. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  26. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  27. 2014Learning with Pseudo-Ensemblescited by 241
  28. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  29. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  30. 2018Semi-Supervised Sequence Modeling with Cross-View Trainingcited by 75

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 19; 2024: 14; 2025: 13; 2026: 5.

66 papers.

Sort:

By area:

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  3. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  4. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  5. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  6. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Modelscited by 117
  7. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  8. 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 57
  9. 2022Language Models in the Loop: Incorporating Prompting into Weak Supervisioncited by 51
  10. 2023Improving Diffusion Models for Scene Text Editing with Dual Encoderscited by 44
  11. 2023Benchmarking and Improving Generator-Validator Consistency of Language Modelscited by 43
  12. 2024Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justificationscited by 41
38 more
  1. 2022Measuring Reliability of Large Language Models through Semantic Consistencycited by 38
  2. 2023Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMscited by 36
  3. 2024On the Worst Prompt Performance of Large Language Modelscited by 36
  4. 2023Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizercited by 35
  5. 2024PURPLE: Making a Large Language Model a Better SQL Writercited by 34
  6. 2022Doubly Right Object Recognition: A Why Prompt for Visual Rationalescited by 29
  7. 2023Predicting Question-Answering Performance of Large Language Models through Semantic Consistencycited by 28
  8. 2022Zero-Shot Text Classification with Self-Trainingcited by 26
  9. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  10. 2023Knowledge-Augmented Language Model Verificationcited by 21
  11. 2024A Survey on the Honesty of Large Language Modelscited by 21
  12. 2023Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verificationcited by 19
  13. 2024On the Consistency of Video Large Language Models in Temporal Comprehensioncited by 18
  14. 2022Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalizationcited by 15
  15. 2023Semantic Consistency for Assuring Reliability of Large Language Modelscited by 15
  16. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  17. 2025Enhancing Semantic Consistency of Large Language Models through Model Editing: An Interpretability-Oriented Approachcited by 14
  18. 2022Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Modelscited by 13
  19. 2023SummQA at MEDIQA-Chat 2023:In-Context Learning with GPT-4 for Medical Summarizationcited by 13
  20. 2025Improving Consistency in Large Language Models through Chain of Guidancecited by 13
  21. 2024SQL-to-Schema Enhances Schema Linking in Text-to-SQLcited by 12
  22. 2023Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Modelscited by 11
  23. 2025ContextNav: Towards Agentic Multimodal In-Context Learningcited by 9
  24. 2022Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Taskscited by 8
  25. 2023How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Rankingcited by 8
  26. 2024A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarioscited by 8
  27. 2023Test-Time Self-Adaptive Small Language Models for Question Answeringcited by 7
  28. 2025LF-Steering: Latent Feature Activation Steering for Enhancing Semantic Consistency in Large Language Modelscited by 7
  29. 2023Prompt to be Consistent is Better than Self-Consistent? Few-Shot and Zero-Shot Fact Verification with Pre-trained Language Modelscited by 6
  30. 2023diff History for Neural Language Agentscited by 6
  31. 2024Latent Space Interpretation for Stylistic Analysis and Explainable Authorship Attributioncited by 6
  32. 2022LANIT: Language-Driven Image-to-Image Translation for Unlabeled Datacited by 5
  33. 2023Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Promptingcited by 5
  34. 2025Same Question, Different Words: A Latent Adversarial Framework for Prompt Robustnesscited by 5
  35. 2025Whose Name Comes Up? Auditing LLM-Based Scholar Recommendationscited by 5
  36. 2025Prompt Candidates, then Distill: A Teacher-Student Framework for LLM-driven Data Annotationcited by 5
  37. 2022Zero-Label Prompt Selectioncited by 4
  38. 2023Improving Language Models Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionarycited by 4

50 most cited of 66 All 66 (JSON)