# Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations

arXiv: https://arxiv.org/abs/2310.07849 · HTML: https://mlsanity.com/paper/2310.07849/

## Key facts

- Cited by 100 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 32 of its 59 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Is synthetic data from generative models ready for image recognition?](https://mlsanity.com/paper/2210.07574/) (2022)
- [Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts](https://mlsanity.com/paper/1307.5336/) (2013)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort](https://mlsanity.com/paper/2104.06490/) (2021)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [Does Synthetic Data Generation of LLMs Help Clinical Text Mining?](https://mlsanity.com/paper/2303.04360/) (2023)
- [Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://mlsanity.com/paper/2306.04140/) (2023)
- [Jury Learning: Integrating Dissenting Voices into Machine Learning Models](https://mlsanity.com/paper/2202.02950/) (2022)
- [FewRel 2.0: Towards More Challenging Few-Shot Relation Classification](https://mlsanity.com/paper/1910.07124/) (2019)
- [Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text](https://mlsanity.com/paper/2107.01294/) (2021)
- [Towards Zero-Label Language Learning](https://mlsanity.com/paper/2109.09193/) (2021)
- [Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation](https://mlsanity.com/paper/2205.00501/) (2022)
- [This dataset does not exist: training models from generated images](https://mlsanity.com/paper/1911.02888/) (2019)
- [On the Creativity of Large Language Models](https://mlsanity.com/paper/2304.00008/) (2023)
- [Data Augmentation for Intent Classification with Off-the-shelf Large Language Models](https://mlsanity.com/paper/2204.01959/) (2022)
- [Weakly Supervised Data Augmentation Through Prompting for Dialogue Understanding](https://mlsanity.com/paper/2210.14169/) (2022)

## Built on by (most cited first, 30 of 100)

- [RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2407.11005/) (2024)
- [Synthetic Data Generation Using Large Language Models: Advances in Text and Code](https://mlsanity.com/paper/2503.14023/) (2025)
- [Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective](https://mlsanity.com/paper/2406.00793/) (2024)
- [1.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Training](https://mlsanity.com/paper/2503.19633/) (2025)
- [Can LLMs Replace Manual Annotation of Software Engineering Artifacts?](https://mlsanity.com/paper/2408.05534/) (2024)
- [MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model](https://mlsanity.com/paper/2409.07486/) (2024)
- [Federated Large Language Models: Current Progress and Future Directions](https://mlsanity.com/paper/2409.15723/) (2024)
- [Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions](https://mlsanity.com/paper/2504.14772/) (2025)
- [Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detection](https://mlsanity.com/paper/2403.13335/) (2024)
- [TarGEN: Targeted Data Generation with Large Language Models](https://mlsanity.com/paper/2310.17876/) (2023)
- [JITServe: SLO-aware LLM Serving with Imprecise Request Information](https://mlsanity.com/paper/2504.20068/) (2025)
- [Differentially Private Tabular Data Synthesis using Large Language Models](https://mlsanity.com/paper/2406.01457/) (2024)
- [The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions](https://mlsanity.com/paper/2406.12480/) (2024)
- [Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective](https://mlsanity.com/paper/2410.01720/) (2024)
- [A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone](https://mlsanity.com/paper/2505.12781/) (2025)
- [CTSyn: A Foundation Model for Cross Tabular Data Generation](https://mlsanity.com/paper/2406.04619/) (2024)
- [The New Agronomists: Language Models are Experts in Crop Management](https://mlsanity.com/paper/2403.19839/) (2024)
- [Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models](https://mlsanity.com/paper/2406.12397/) (2024)
- [Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding](https://mlsanity.com/paper/2404.04293/) (2024)
- [Synthetic Text Generation for Training Large Language Models via Gradient Matching](https://mlsanity.com/paper/2502.17607/) (2025)
- [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2503.16585/) (2025)
- [DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models](https://mlsanity.com/paper/2411.03250/) (2024)
- [A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models](https://mlsanity.com/paper/2404.14445/) (2024)
- [Evaluating Explanations Through LLMs: Beyond Traditional User Studies](https://mlsanity.com/paper/2410.17781/) (2024)
- [Synthetic Data Generation with Large Language Models for Personalized Community Question Answering](https://mlsanity.com/paper/2410.22182/) (2024)
- [WhatELSE: Shaping Narrative Spaces at Configurable Level of Abstraction for AI-bridged Interactive Storytelling](https://mlsanity.com/paper/2502.18641/) (2025)
- [FM2DS: Few-Shot Multimodal Multihop Data Synthesis with Knowledge Distillation for Question Answering](https://mlsanity.com/paper/2412.07030/) (2024)
- [Automatic Dataset Generation for Knowledge Intensive Question Answering Tasks](https://mlsanity.com/paper/2505.14212/) (2025)
- [Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classification](https://mlsanity.com/paper/2410.21526/) (2024)
- [Can Code-Switched Texts Activate a Knowledge Switch in LLMs? A Case Study on English-Korean Code-Switching](https://mlsanity.com/paper/2410.18436/) (2024)

## Abstract (arXiv)

The collection and curation of high-quality training data is crucial for developing text classification models with superior performance, but it is often associated with significant costs and time investment. Researchers have recently explored using large language models (LLMs) to generate synthetic datasets as an alternative approach. However, the effectiveness of the LLM-generated synthetic data in supporting model training is inconsistent across different classification tasks. To better understand factors that moderate the effectiveness of the LLM-generated synthetic data, in this study, we look into how the performance of models trained on these synthetic data may vary with the subjectivity of classification. Our results indicate that subjectivity, at both the task level and instance level, is negatively associated with the performance of the model trained on synthetic data. We conclude by discussing the implications of our work on the potential and limitations of leveraging LLM for synthetic data generation.