# ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval

arXiv: https://arxiv.org/abs/2305.10703 · HTML: https://mlsanity.com/paper/2305.10703/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 103 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 30 of 74)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)

## Built on by (most cited first, 26 of 26)

- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [Synthetic Data Generation Using Large Language Models: Advances in Text and Code](https://mlsanity.com/paper/2503.14023/) (2025)
- [Generative AI for Synthetic Data Generation: Methods, Challenges and the Future](https://mlsanity.com/paper/2403.04190/) (2024)
- [TarGEN: Targeted Data Generation with Large Language Models](https://mlsanity.com/paper/2310.17876/) (2023)
- [Retrieval-style In-Context Learning for Few-shot Hierarchical Text Classification](https://mlsanity.com/paper/2406.17534/) (2024)
- [SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation](https://mlsanity.com/paper/2405.10040/) (2024)
- [DyGen: Learning from Noisy Labels via Dynamics-Enhanced Generative Modeling](https://mlsanity.com/paper/2305.19395/) (2023)
- [It's All Relative! -- A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Prediction](https://mlsanity.com/paper/2311.07930/) (2023)
- [Few-shot LLM Synthetic Data with Distribution Matching](https://mlsanity.com/paper/2502.08661/) (2025)
- [Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation](https://mlsanity.com/paper/2407.16008/) (2024)
- [A Little Human Data Goes A Long Way](https://mlsanity.com/paper/2410.13098/) (2024)
- [Retrieval Augmented Zero-Shot Text Classification](https://mlsanity.com/paper/2406.15241/) (2024)
- [Training Task Experts through Retrieval Based Distillation](https://mlsanity.com/paper/2407.05463/) (2024)
- [CorrSynth -- A Correlated Sampling Method for Diverse Dataset Generation from LLMs](https://mlsanity.com/paper/2411.08553/) (2024)
- [Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review](https://mlsanity.com/paper/2510.17892/) (2025)
- [Text Classification Under Class Distribution Shift: A Survey](https://mlsanity.com/paper/2502.12965/) (2025)
- [Does Prompt Design Impact Quality of Data Imputation by LLMs?](https://mlsanity.com/paper/2506.04172/) (2025)
- [XAL: EXplainable Active Learning Makes Classifiers Better Low-resource Learners](https://mlsanity.com/paper/2310.05502/) (2023)
- [ProgGen: Generating Named Entity Recognition Datasets Step-by-step with Self-Reflexive Large Language Models](https://mlsanity.com/paper/2403.11103/) (2024)
- [Clustering Algorithms and RAG Enhancing Semi-Supervised Text Classification with Large LLMs](https://mlsanity.com/paper/2411.06175/) (2024)
- [Local Boosting for Weakly-Supervised Learning](https://mlsanity.com/paper/2306.02859/) (2023)
- [Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language Models](https://mlsanity.com/paper/2406.09206/) (2024)
- [JAPAGEN: Efficient Few/Zero-shot Learning via Japanese Training Dataset Generation with LLM](https://mlsanity.com/paper/2412.06738/) (2024)
- [Batch Aggregation: An Approach to Enhance Text Classification with Correlated Augmented Data](https://mlsanity.com/paper/2504.05020/) (2025)
- [MOSLD-Bench: Multilingual Open-Set Learning and Discovery Benchmark for Text Categorization](https://mlsanity.com/paper/2601.13437/) (2026)

## Abstract (arXiv)

With the development of large language models (LLMs), zero-shot learning has attracted much attention for various NLP tasks. Different from prior works that generate training data with billion-scale natural language generation (NLG) models, we propose a retrieval-enhanced framework to create training data from a general-domain unlabeled corpus. To realize this, we first conduct contrastive pretraining to learn an unsupervised dense retriever for extracting the most relevant documents using class-descriptive verbalizers. We then further propose two simple strategies, namely Verbalizer Augmentation with Demonstrations and Self-consistency Guided Filtering to improve the topic coverage of the dataset while removing noisy examples. Experiments on nine datasets demonstrate that REGEN achieves 4.3% gain over the strongest baselines and saves around 70% of the time compared to baselines using large NLG models. Besides, REGEN can be naturally integrated with recently proposed large language models to boost performance.