mlsanityData to 14 Oct 2026

Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models

arXiv 2310.13671 ↗First posted 20 Oct 2023cs.CL cs.AIcited by 30

Not on the map. Why?

Abstract (arXiv)

*Data Synthesis* is a promising way to train a small model with very little labeled data. One approach for data synthesis is to leverage the rich knowledge from large language models to synthesize pseudo training examples for small models, making it possible to achieve both data and compute efficiency at the same time. However, a key challenge in data synthesis is that the synthesized dataset often suffers from a large distributional discrepancy from the *real task* data distribution. Thus, in this paper, we propose *Synthesis Step by Step* (**S3**), a data synthesis framework that shrinks this distribution gap by iteratively extrapolating the errors made by a small model trained on the synthesized dataset on a small real-world validation dataset using a large language model. Extensive experiments on multiple NLP tasks show that our approach improves the performance of a small model by reducing the gap between the synthetic dataset and the real data, resulting in significant improvement compared to several baselines: 9.48% improvement compared to ZeroGen and 2.73% compared to GoldGen, and at most 15.17% improvement compared to the small model trained on human-annotated data.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 30 of its 49 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

30 of 49 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
18 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  2. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  3. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  4. 2021Knowledge distillation: A good teacher is patient and consistentcited by 259
  5. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  6. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  7. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  8. 2019Not Enough Data? Deep Learning to the Rescue!cited by 175
  9. 2021Meta-learning via Language Model In-context Tuningcited by 164
  10. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  11. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  12. 2017Dataset Augmentation in Feature Spacecited by 124
  13. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 113
  14. 2020Training Question Answering Models From Synthetic Datacited by 95
  15. 2021Data Augmentation Approaches in Natural Language Processing: A Surveycited by 82
  16. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  17. 2021A Survey on Green Deep Learningcited by 52
  18. 2022Language Models in the Loop: Incorporating Prompting into Weak Supervisioncited by 52

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 15; 2025: 9; 2026: 3.

30 papers.

Sort:

By area:

  1. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  2. 2023Learning From Mistakes Makes LLM Better Reasonercited by 101
  3. 2024TheoremLlama: Transforming General-Purpose LLMs into Lean4 Expertscited by 59
  4. 2024CultureLLM: Incorporating Cultural Differences into Large Language Modelscited by 50
  5. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 45
  6. 2025MA-LoT: Model-Collaboration Lean-based Long Chain-of-Thought Reasoning enhances Formal Theorem Provingcited by 24
  7. 2023Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Modelscited by 20
  8. 2024SynthesizRR: Generating Diverse Datasets with Retrieval Augmentationcited by 13
  9. 2024Piecing It All Together: Verifying Multi-Hop Multimodal Claimscited by 12
  10. 2023Multistage Collaborative Knowledge Distillation from a Large Language Model for Semi-Supervised Sequence Generationcited by 10
  11. 2025Few-shot LLM Synthetic Data with Distribution Matchingcited by 10
  12. 2024TAIA: Large Language Models are Out-of-Distribution Data Learnerscited by 8
18 more
  1. 2024LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to Small-Scale Local LLMscited by 8
  2. 2024DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generationcited by 8
  3. 2025TreeSynth: Synthesizing Diverse Data from Scratch via Tree-Guided Subspace Partitioningcited by 7
  4. 2025Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learningcited by 7
  5. 2025GAR: Generative Adversarial Reinforcement Learning for Formal Theorem Provingcited by 6
  6. 2025Automatic Labelling with Open-source LLMs using Dynamic Label Schema Integrationcited by 5
  7. 2025FANS -- Formal Answer Selection for Natural Language Math Reasoning Using Lean4cited by 5
  8. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  9. 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3
  10. 2024Training Task Experts through Retrieval Based Distillationcited by 3
  11. 2024CorrSynth -- A Correlated Sampling Method for Diverse Dataset Generation from LLMscited by 3
  12. 2024RED-CT: A Systems Design Methodology for Using LLM-labeled Data to Train and Deploy Edge Classifiers for Computational Social Sciencecited by 2
  13. 2024Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval-Augmented Generationcited by 2
  14. 2026Pioneer Agent: Continual Improvement of Small Language Models in Productioncited by 2
  15. 2024Enhancing SLM via ChatGPT and Dataset Augmentationcited by 0
  16. 2025ARISE: Iterative Rule Induction and Synthetic Data Generation for Text Classificationcited by 0
  17. 2026Entropy-Based Data Selection for Language Modelscited by 0
  18. 2026Training-Aware Target Coverage for Synthetic Data Selectioncited by 0