# Zero-Shot Text Classification with Self-Training

arXiv: https://arxiv.org/abs/2210.17541 · HTML: https://mlsanity.com/paper/2210.17541/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 21 of its 42 references that are in the map.
- First posted on arXiv 31 Oct 2022.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-training for few-shot NLP](https://mlsanity.com/synth/self-training-for-few-shot-nlp/)

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Confidence Regularized Self-Training](https://mlsanity.com/paper/1908.09822/) (2019)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks](https://mlsanity.com/paper/1811.01088/) (2018)
- [Muppet: Massive Multi-task Representations with Pre-Finetuning](https://mlsanity.com/paper/2101.11038/) (2021)
- [Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections](https://mlsanity.com/paper/2104.04670/) (2021)
- [Self-training Improves Pre-training for Natural Language Understanding](https://mlsanity.com/paper/2010.02194/) (2020)
- [FLEX: Unifying Evaluation for Few-Shot NLP](https://mlsanity.com/paper/2107.07170/) (2021)
- [Prompt Consistency for Zero-Shot Task Generalization](https://mlsanity.com/paper/2205.00049/) (2022)
- [Weakly-supervised Text Classification Based on Keyword Graph](https://mlsanity.com/paper/2110.02591/) (2021)
- [Ask2Transformers: Zero-Shot Domain labelling with Pre-trained Language Models](https://mlsanity.com/paper/2101.02661/) (2021)

## Built on by (most cited first, 26 of 26)

- [Language Models for Text Classification: Is In-Context Learning Enough?](https://mlsanity.com/paper/2403.17661/) (2024)
- [ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval](https://mlsanity.com/paper/2305.10703/) (2023)
- [Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner](https://mlsanity.com/paper/2305.01711/) (2023)
- [Rethinking Semi-supervised Learning with Language Models](https://mlsanity.com/paper/2305.13002/) (2023)
- [Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLM](https://mlsanity.com/paper/2304.11872/) (2023)
- [Lessons from the Use of Natural Language Inference (NLI) in Requirements Engineering Tasks](https://mlsanity.com/paper/2405.05135/) (2024)
- [ZARA: Improving Few-Shot Self-Rationalization for Small Language Models](https://mlsanity.com/paper/2305.07355/) (2023)
- [LLM-Generated Natural Language Meets Scaling Laws: New Explorations and Data Augmentation Methods](https://mlsanity.com/paper/2407.00322/) (2024)
- [A Simple and Effective Framework for Strict Zero-Shot Hierarchical Classification](https://mlsanity.com/paper/2305.15282/) (2023)
- [Zero-Shot Text Classification via Self-Supervised Tuning](https://mlsanity.com/paper/2305.11442/) (2023)
- [A Benchmark on Extremely Weakly Supervised Text Classification: Reconcile Seed Matching and Prompting Approaches](https://mlsanity.com/paper/2305.12749/) (2023)
- [QAID: Question Answering Inspired Few-shot Intent Detection](https://mlsanity.com/paper/2303.01593/) (2023)
- [The Benefits of Label-Description Training for Zero-Shot Text Classification](https://mlsanity.com/paper/2305.02239/) (2023)
- [TEXT2DB: Integration-Aware Information Extraction with Large Language Model Agents](https://mlsanity.com/paper/2510.24014/) (2025)
- [Enhancing Self-Correction in Large Language Models through Multi-Perspective Reflection](https://mlsanity.com/paper/2601.07780/) (2026)
- [Empowering Sentence Encoders with Prompting and Label Retrieval for Zero-shot Text Classification](https://mlsanity.com/paper/2212.10391/) (2022)
- [WC-SBERT: Zero-Shot Text Classification via SBERT with Self-Training for Wikipedia Categories](https://mlsanity.com/paper/2307.15293/) (2023)
- [Open-world Multi-label Text Classification with Extremely Weak Supervision](https://mlsanity.com/paper/2407.05609/) (2024)
- [Interactive Machine Teaching by Labeling Rules and Instances](https://mlsanity.com/paper/2409.05199/) (2024)
- [Text Classification Under Class Distribution Shift: A Survey](https://mlsanity.com/paper/2502.12965/) (2025)
- [Medifact at PerAnsSumm 2025: Leveraging Lightweight Models for Perspective-Specific Summarization of Clinical Q&A Forums](https://mlsanity.com/paper/2503.16513/) (2025)
- [TK-KNN: A Balanced Distance-Based Pseudo Labeling Approach for Semi-Supervised Intent Classification](https://mlsanity.com/paper/2310.11607/) (2023)
- [Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language Models](https://mlsanity.com/paper/2406.09206/) (2024)
- [Analysis of Socially Unacceptable Discourse with Zero-shot Learning](https://mlsanity.com/paper/2409.13735/) (2024)
- [MOSLD-Bench: Multilingual Open-Set Learning and Discovery Benchmark for Text Categorization](https://mlsanity.com/paper/2601.13437/) (2026)
- [DECSELFMASK: Leveraging Unlabeled Text via Self-Relevance-Guided Masking for Decoder-Only Classification](https://mlsanity.com/paper/2606.09466/) (2026)

## Abstract (arXiv)

Recent advances in large pretrained language models have increased attention to zero-shot text classification. In particular, models finetuned on natural language inference datasets have been widely adopted as zero-shot classifiers due to their promising results and off-the-shelf availability. However, the fact that such models are unfamiliar with the target task can lead to instability and performance issues. We propose a plug-and-play method to bridge this gap using a simple self-training approach, requiring only the class names along with an unlabeled dataset, and without the need for domain expertise or trial and error. We show that fine-tuning the zero-shot classifier on its most confident predictions leads to significant performance gains across a wide range of text classification tasks, presumably since self-training adapts the zero-shot model to the task at hand.