# Zero-shot Text Classification With Generative Language Models

arXiv: https://arxiv.org/abs/1912.10165 · HTML: https://mlsanity.com/paper/1912.10165/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2019.

## Built on by (most cited first, 30 of 55)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [How Many Data Points is a Prompt Worth?](https://mlsanity.com/paper/2103.08493/) (2021)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [Entailment as Few-Shot Learner](https://mlsanity.com/paper/2104.14690/) (2021)
- [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://mlsanity.com/paper/2109.03910/) (2021)
- [Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction](https://mlsanity.com/paper/2109.03659/) (2021)
- [RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction](https://mlsanity.com/paper/2203.09101/) (2022)
- [TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models](https://mlsanity.com/paper/2407.09424/) (2024)
- [True Few-Shot Learning with Prompts -- A Real-World Perspective](https://mlsanity.com/paper/2111.13440/) (2021)
- [Understanding Telecom Language Through Large Language Models](https://mlsanity.com/paper/2306.07933/) (2023)
- [On Robust Prefix-Tuning for Text Classification](https://mlsanity.com/paper/2203.10378/) (2022)
- [GLiClass: Generalist Lightweight Model for Sequence Classification Tasks](https://mlsanity.com/paper/2508.07662/) (2025)
- [Automated Essay Scoring Using Transformer Models](https://mlsanity.com/paper/2110.06874/) (2021)
- [VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation](https://mlsanity.com/paper/2403.12415/) (2024)
- [RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Framework](https://mlsanity.com/paper/2410.19109/) (2024)
- [Match-Prompt: Improving Multi-task Generalization Ability for Neural Text Matching via Prompt Learning](https://mlsanity.com/paper/2204.02725/) (2022)
- [Towards Few-shot Entity Recognition in Document Images: A Label-aware Sequence-to-Sequence Framework](https://mlsanity.com/paper/2204.05819/) (2022)
- [Elaboration-Generating Commonsense Question Answering at Scale](https://mlsanity.com/paper/2209.01232/) (2022)
- [ChatGPT for Suicide Risk Assessment on Social Media: Quantitative Evaluation of Model Performance, Potentials and Limitations](https://mlsanity.com/paper/2306.09390/) (2023)
- [Enhancing Trust in LLMs: Algorithms for Comparing and Interpreting LLMs](https://mlsanity.com/paper/2406.01943/) (2024)
- [Semi-Supervised Exaggeration Detection of Health Science Press Releases](https://mlsanity.com/paper/2108.13493/) (2021)
- [Large Language Models for Propaganda Detection](https://mlsanity.com/paper/2310.06422/) (2023)
- [NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillation](https://mlsanity.com/paper/2312.05979/) (2023)
- [Pre-trained Token-replaced Detection Model as Few-shot Learner](https://mlsanity.com/paper/2203.03235/) (2022)
- [Framing the News:From Human Perception to Large Language Model Inferences](https://mlsanity.com/paper/2304.14456/) (2023)
- [Beyond the Next Token: Towards Prompt-Robust Zero-Shot Classification via Efficient Multi-Token Prediction](https://mlsanity.com/paper/2504.03159/) (2025)

## Abstract (arXiv)

This work investigates the use of natural language to enable zero-shot model adaptation to new tasks. We use text and metadata from social commenting platforms as a source for a simple pretraining task. We then provide the language model with natural language descriptions of classification tasks as input and train it to generate the correct answer in natural language via a language modeling objective. This allows the model to generalize to new classification tasks without the need for multiple multitask classification heads. We show the zero-shot performance of these generative language models, trained with weak supervision, on six benchmark text classification datasets from the torchtext library. Despite no access to training data, we achieve up to a 45% absolute improvement in classification accuracy over random or majority class baselines. These results show that natural language can serve as simple and powerful descriptors for task adaptation. We believe this points the way to new metalearning strategies for text problems.