mlsanityData to 7 Oct 2026

Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models

arXiv 2310.01119 ↗First posted 2 Oct 2023cs.CL cs.LGcited by 3

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

The in-context learning ability of large language models (LLMs) enables them to generalize to novel downstream tasks with relatively few labeled examples. However, they require enormous computational resources to be deployed. Alternatively, smaller models can solve specific tasks if fine-tuned with enough labeled examples. These examples, however, are expensive to obtain. In pursuit of the best of both worlds, we study synthetic data generation of fine-tuning training data via fine-tuned teacher LLMs to improve the downstream performance of much smaller models. In four text classification and two text generation tasks, we find that both data generation and annotation dramatically improve the respective downstream model's performance, occasionally necessitating only a minor fraction of the original training dataset.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 32 of its 38 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

32 of 38 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
20 more
  1. 2023Textbooks Are All You Needcited by 561
  2. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  3. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  4. 2022Large Language Models Are Reasoning Teacherscited by 318
  5. 2023Challenges and Applications of Large Language Modelscited by 251
  6. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  7. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  8. 2022Is GPT-3 a Good Data Annotator?cited by 208
  9. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  10. 2015Semi-supervised Sequence Learningcited by 198
  11. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  12. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  13. 2021Generating Datasets with Pretrained Language Modelscited by 192
  14. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  15. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  16. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  17. 2022AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversationcited by 79
  18. 2022Data Augmentation for Intent Classification with Off-the-shelf Large Language Modelscited by 49
  19. 2022Weakly Supervised Data Augmentation Through Prompting for Dialogue Understandingcited by 32
  20. 2022Explicit Knowledge Transfer for Weakly-Supervised Code Generationcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024MoDEM: Mixture of Domain Expert Modelscited by 9
  2. 2025A Comprehensive Dataset for Human vs. AI Generated Text Detectioncited by 3
  3. 2025Modelling and Classifying the Components of a Literature Reviewcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  2. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  3. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification
  4. 2025Towards Active Synthetic Data Generation for Finetuning Language Models
  5. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  6. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  7. 2025Scaling Laws of Synthetic Data for Language Models
  8. 2024Evaluating Language Models as Synthetic Data Generators