# The Benefits of Label-Description Training for Zero-Shot Text Classification

arXiv: https://arxiv.org/abs/2305.02239 · HTML: https://mlsanity.com/paper/2305.02239/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 3 May 2023.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs](https://mlsanity.com/paper/1803.08035/) (2018)
- [Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence](https://mlsanity.com/paper/1903.09588/) (2019)
- [Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification](https://mlsanity.com/paper/2010.13641/) (2020)
- [True Few-Shot Learning with Prompts -- A Real-World Perspective](https://mlsanity.com/paper/2111.13440/) (2021)
- [Integrating Semantic Knowledge to Tackle Zero-shot Text Classification](https://mlsanity.com/paper/1903.12626/) (2019)
- [Description Based Text Classification with Reinforcement Learning](https://mlsanity.com/paper/2002.03067/) (2020)
- [Zero-Shot Text Classification with Self-Training](https://mlsanity.com/paper/2210.17541/) (2022)
- [Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models](https://mlsanity.com/paper/2210.14803/) (2022)
- [Beyond Prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations](https://mlsanity.com/paper/2210.16637/) (2022)
- [Unsupervised Label Refinement Improves Dataless Text Classification](https://mlsanity.com/paper/2012.04194/) (2020)
- [NatCat: Weakly Supervised Text Classification with Naturally Annotated Resources](https://mlsanity.com/paper/2009.14335/) (2020)
- [Unsupervised Ranking and Aggregation of Label Descriptions for Zero-Shot Classifiers](https://mlsanity.com/paper/2204.09481/) (2022)

## Built on by (most cited first, 5 of 5)

- [Grasping the Essentials: Tailoring Large Language Models for Zero-Shot Relation Extraction](https://mlsanity.com/paper/2402.11142/) (2024)
- [Meta-Tuning LLMs to Leverage Lexical Knowledge for Generalizable Language Style Understanding](https://mlsanity.com/paper/2305.14592/) (2023)
- [Using Large Language Models to Automate Category and Trend Analysis of Scientific Articles: An Application in Ophthalmology](https://mlsanity.com/paper/2308.16688/) (2023)
- [Exploring Description-Augmented Dataless Intent Classification](https://mlsanity.com/paper/2407.17862/) (2024)
- [Label-Centric Knowledge Retrieval for Zero-Shot Text Classification](https://mlsanity.com/paper/2610.16182/) (2026)

## Abstract (arXiv)

Pretrained language models have improved zero-shot text classification by allowing the transfer of semantic knowledge from the training data in order to classify among specific label sets in downstream tasks. We propose a simple way to further improve zero-shot accuracies with minimal effort. We curate small finetuning datasets intended to describe the labels for a task. Unlike typical finetuning data, which has texts annotated with labels, our data simply describes the labels in language, e.g., using a few related terms, dictionary/encyclopedia entries, and short templates. Across a range of topic and sentiment datasets, our method is more accurate than zero-shot by 17-19% absolute. It is also more robust to choices required for zero-shot classification, such as patterns for prompting the model to classify and mappings from labels to tokens in the model's vocabulary. Furthermore, since our data merely describes the labels but does not use input texts, finetuning on it yields a model that performs strongly on multiple text domains for a given label set, even improving over few-shot out-of-domain classification in multiple settings.