# Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLM

arXiv: https://arxiv.org/abs/2304.11872 · HTML: https://mlsanity.com/paper/2304.11872/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 20 of its 42 references that are in the map.
- First posted on arXiv 24 Apr 2023.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-training for few-shot NLP](https://mlsanity.com/synth/self-training-for-few-shot-nlp/)

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [Distilling Reasoning Capabilities into Smaller Language Models](https://mlsanity.com/paper/2212.00193/) (2022)
- [Contrastive Distillation on Intermediate Representations for Language Model Compression](https://mlsanity.com/paper/2009.14167/) (2020)
- [Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learning](https://mlsanity.com/paper/2103.07552/) (2021)
- [Zero-Shot Text Classification with Self-Training](https://mlsanity.com/paper/2210.17541/) (2022)
- [Weakly-supervised Text Classification Based on Keyword Graph](https://mlsanity.com/paper/2110.02591/) (2021)
- [Beyond Prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representations](https://mlsanity.com/paper/2210.16637/) (2022)

## Built on by (most cited first, 12 of 12)

- [LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Models](https://mlsanity.com/paper/2309.09506/) (2023)
- [Zero-shot Generative Large Language Models for Systematic Review Screening Automation](https://mlsanity.com/paper/2401.06320/) (2024)
- [Learning on Graphs with Large Language Models(LLMs): A Deep Dive into Model Robustness](https://mlsanity.com/paper/2407.12068/) (2024)
- [Recognizing Emotion Regulation Strategies from Human Behavior with Large Language Models](https://mlsanity.com/paper/2408.04420/) (2024)
- [WC-SBERT: Zero-Shot Text Classification via SBERT with Self-Training for Wikipedia Categories](https://mlsanity.com/paper/2307.15293/) (2023)
- [Multi-label Sequential Sentence Classification via Large Language Model](https://mlsanity.com/paper/2411.15623/) (2024)
- [Cross-domain Chinese Sentence Pattern Parsing](https://mlsanity.com/paper/2402.16311/) (2024)
- [Small sample-based adaptive text classification through iterative and contrastive description refinement](https://mlsanity.com/paper/2508.00957/) (2025)
- [RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty](https://mlsanity.com/paper/2602.12424/) (2026)
- [Semantic Reranking at Inference Time for Hard Examples in Rhetorical Role Labeling](https://mlsanity.com/paper/2605.18007/) (2026)
- [VeriCam: A Verification Baseline for the Classification of Unknown Data](https://mlsanity.com/paper/2608.31107/) (2026)
- [Zero-Shot Visualization: Exploring Text Corpora with User-Prompted Axes](https://mlsanity.com/paper/2610.06889/) (2026)

## Abstract (arXiv)

The remarkable performance of large language models (LLMs) in zero-shot language understanding has garnered significant attention. However, employing LLMs for large-scale inference or domain-specific fine-tuning requires immense computational resources due to their substantial model size. To overcome these limitations, we introduce a novel method, namely GenCo, which leverages the strong generative power of LLMs to assist in training a smaller and more adaptable language model. In our method, an LLM plays an important role in the self-training loop of a smaller model in two important ways. Firstly, the LLM is used to augment each input instance with a variety of possible continuations, enriching its semantic context for better understanding. Secondly, it helps crafting additional high-quality training pairs, by rewriting input texts conditioned on predicted labels. This ensures the generated texts are highly relevant to the predicted labels, alleviating the prediction error during pseudo-labeling, while reducing the dependency on large volumes of unlabeled text. In our experiments, GenCo outperforms previous state-of-the-art methods when only limited (<5% of original) in-domain text data is available. Notably, our approach surpasses the performance of Alpaca-7B with human prompts, highlighting the potential of leveraging LLM for self-training.