# Incubating Text Classifiers Following User Instruction with Nothing but LLM

arXiv: https://arxiv.org/abs/2404.10877 · HTML: https://mlsanity.com/paper/2404.10877/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 59 references that are in the map.
- First posted on arXiv 16 Apr 2024.

## Builds on (most cited first, 30 of 39)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step](https://mlsanity.com/paper/2306.14050/) (2023)
- [MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages](https://mlsanity.com/paper/2204.08582/) (2022)
- [Text Classification via Large Language Models](https://mlsanity.com/paper/2305.08377/) (2023)
- [Evolution of Semantic Similarity -- A Survey](https://mlsanity.com/paper/2004.13820/) (2020)
- [ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback](https://mlsanity.com/paper/2210.12329/) (2022)
- [X-Class: Text Classification with Extremely Weak Supervision](https://mlsanity.com/paper/2010.12794/) (2020)
- [Pre-trained Language Models Can be Fully Zero-Shot Learners](https://mlsanity.com/paper/2212.06950/) (2022)
- [BFClass: A Backdoor-free Text Classification Framework](https://mlsanity.com/paper/2109.10855/) (2021)
- [AUGNLG: Few-shot Natural Language Generation using Self-trained Data Augmentation](https://mlsanity.com/paper/2106.05589/) (2021)
- [WeDef: Weakly Supervised Backdoor Defense for Text Classification](https://mlsanity.com/paper/2205.11803/) (2022)
- [A Brief Survey of Text Mining: Classification, Clustering and Extraction Techniques](https://mlsanity.com/paper/1707.02919/) (2017)
- [LOPS: Learning Order Inspired Pseudo-Label Selection for Weakly Supervised Text Classification](https://mlsanity.com/paper/2205.12528/) (2022)

## Built on by (most cited first, 6 of 6)

- [SelfCheck-Eval: A Multi-Module Framework for Zero-Resource Hallucination Detection in Large Language Models](https://mlsanity.com/paper/2502.01812/) (2025)
- [Text Grafting: Near-Distribution Weak Supervision for Minority Classes in Text Classification](https://mlsanity.com/paper/2406.11115/) (2024)
- [Correlation and Navigation in the Vocabulary Key Representation Space of Language Models](https://mlsanity.com/paper/2410.02284/) (2024)
- [Semantic Reranking at Inference Time for Hard Examples in Rhetorical Role Labeling](https://mlsanity.com/paper/2605.18007/) (2026)
- [Four Ledgers, Not One Score: Responsible Communication of LLM-Judge Calibration in Biomedical ML](https://mlsanity.com/paper/2609.15015/) (2026)
- [High-Value Synthetic Supervision for Parameter-Efficient Adaptation of a Compact Japanese Speech Model](https://mlsanity.com/paper/2610.00026/) (2026)

## Abstract (arXiv)

In this paper, we aim to generate text classification data given arbitrary class definitions (i.e., user instruction), so one can train a small text classifier without any human annotation or raw corpus. Compared with pioneer attempts, our proposed Incubator is the first framework that can handle complicated and even mutually dependent classes (e.g., "TED Talk given by Educator" and "Other"). Specifically, Incubator is an LLM firstly tuned on the instruction-to-data mappings that we obtained from classification datasets and descriptions on HuggingFace together with in-context augmentation by GPT-4. We then refine Incubator by learning on the cluster centers of semantic textual embeddings to emphasize the uniformity and semantic diversity in generations. We compare Incubator on various classification tasks with strong baselines such as direct LLM-based inference and training data generation by prompt engineering. Experiments show Incubator is able to (1) perform well on traditional benchmarks, (2) take label dependency and user preference into consideration, and (3) enable logical text mining by incubating multiple classifiers.