# Pre-trained Language Models Can be Fully Zero-Shot Learners

arXiv: https://arxiv.org/abs/2212.06950 · HTML: https://mlsanity.com/paper/2212.06950/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 14 Dec 2022.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [PTR: Prompt Tuning with Rules for Text Classification](https://mlsanity.com/paper/2105.11259/) (2021)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [Commonsense Knowledge Mining from Pretrained Models](https://mlsanity.com/paper/1909.00505/) (2019)
- [Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification](https://mlsanity.com/paper/2108.02035/) (2021)
- [Text Classification Algorithms: A Survey](https://mlsanity.com/paper/1904.08067/) (2019)
- [OpenPrompt: An Open-source Framework for Prompt-learning](https://mlsanity.com/paper/2111.01998/) (2021)
- [Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models](https://mlsanity.com/paper/2106.13353/) (2021)
- [Zero-shot User Intent Detection via Capsule Neural Networks](https://mlsanity.com/paper/1809.00385/) (2018)
- [Integrating Semantic Knowledge to Tackle Zero-shot Text Classification](https://mlsanity.com/paper/1903.12626/) (2019)
- [Beyond Prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations](https://mlsanity.com/paper/2210.16637/) (2022)

## Built on by (most cited first, 22 of 22)

- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [LumberChunker: Long-Form Narrative Document Segmentation](https://mlsanity.com/paper/2406.17526/) (2024)
- [TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervision](https://mlsanity.com/paper/2403.00165/) (2024)
- [Small Language Models are Good Too: An Empirical Study of Zero-Shot Classification](https://mlsanity.com/paper/2404.11122/) (2024)
- [Universal Self-Adaptive Prompting](https://mlsanity.com/paper/2305.14926/) (2023)
- [A Survey on Retrieval And Structuring Augmented Generation with Large Language Models](https://mlsanity.com/paper/2509.10697/) (2025)
- [A Benchmark on Extremely Weakly Supervised Text Classification: Reconcile Seed Matching and Prompting Approaches](https://mlsanity.com/paper/2305.12749/) (2023)
- [Incubating Text Classifiers Following User Instruction with Nothing but LLM](https://mlsanity.com/paper/2404.10877/) (2024)
- [A LLM-Based Ranking Method for the Evaluation of Automatic Counter-Narrative Generation](https://mlsanity.com/paper/2406.15227/) (2024)
- [Less than One-shot: Named Entity Recognition via Extremely Weak Supervision](https://mlsanity.com/paper/2311.02861/) (2023)
- [SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topics](https://mlsanity.com/paper/2410.01946/) (2024)
- [Debiasing Made State-of-the-art: Revisiting the Simple Seed-based Weak Supervision for Text Classification](https://mlsanity.com/paper/2305.14794/) (2023)
- [Beyond Pairwise: Global Zero-shot Temporal Graph Generation](https://mlsanity.com/paper/2502.11114/) (2025)
- [RulePrompt: Weakly Supervised Text Classification with Prompting PLMs and Self-Iterative Logical Rules](https://mlsanity.com/paper/2403.02932/) (2024)
- [Text Grafting: Near-Distribution Weak Supervision for Minority Classes in Text Classification](https://mlsanity.com/paper/2406.11115/) (2024)
- [Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning](https://mlsanity.com/paper/2404.12897/) (2024)
- [T3: A Novel Zero-shot Transfer Learning Framework Iteratively Training on an Assistant Task for a Target Task](https://mlsanity.com/paper/2409.17640/) (2024)
- [Empirical Study of Code Large Language Models for Binary Security Patch Detection](https://mlsanity.com/paper/2509.06052/) (2025)
- [Zero-shot Retrieval: Augmenting Pre-trained Models with Search Engines](https://mlsanity.com/paper/2311.17949/) (2023)
- [Manual Verbalizer Enrichment for Few-Shot Text Classification](https://mlsanity.com/paper/2410.06173/) (2024)
- [TRepLiNa: Layer-wise CKA+REPINA Alignment Improves Low-Resource Machine Translation in Aya-23 8B](https://mlsanity.com/paper/2510.06249/) (2025)

## Abstract (arXiv)

How can we extend a pre-trained model to many language understanding tasks, without labeled or additional unlabeled data? Pre-trained language models (PLMs) have been effective for a wide range of NLP tasks. However, existing approaches either require fine-tuning on downstream labeled datasets or manually constructing proper prompts. In this paper, we propose nonparametric prompting PLM (NPPrompt) for fully zero-shot language understanding. Unlike previous methods, NPPrompt uses only pre-trained language models and does not require any labeled data or additional raw corpus for further fine-tuning, nor does it rely on humans to construct a comprehensive set of prompt label words. We evaluate NPPrompt against previous major few-shot and zero-shot learning methods on diverse NLP tasks: including text classification, text entailment, similar text retrieval, and paraphrasing. Experimental results demonstrate that our NPPrompt outperforms the previous best fully zero-shot method by big margins, with absolute gains of 12.8% in accuracy on text classification and 18.9% on the GLUE benchmark.