mlsanityData to 14 Oct 2026

Few-shot LLM Synthetic Data with Distribution Matching

arXiv 2502.08661 ↗First posted 9 Feb 2025cs.CL cs.AIcited by 10

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

As large language models (LLMs) advance, their ability to perform in-context learning and few-shot language generation has improved significantly. This has spurred using LLMs to produce high-quality synthetic data to enhance the performance of smaller models like online retrievers or weak LLMs. However, LLM-generated synthetic data often differs from the real data in key language attributes (e.g., styles, tones, content proportions, etc.). As a result, mixing these synthetic data directly with real data may distort the original data distribution, potentially hindering performance improvements. To solve this, we introduce SynAlign: a synthetic data generation and filtering framework based on key attribute distribution matching. Before generation, SynAlign employs an uncertainty tracker surrogated by the Gaussian Process model to iteratively select data clusters distinct from selected ones as demonstrations for new data synthesis, facilitating the efficient exploration diversity of the real data. Then, a latent attribute reasoning method is employed: the LLM summarizes linguistic attributes of demonstrations and then synthesizes new data based on them. This approach facilitates synthesizing diverse data with linguistic attributes that appear in real data.After generation, the Maximum Mean Discrepancy is used as the objective function to learn the sampling weight of each synthetic data, ensuring distribution matching with the real data. Our experiments on multiple text prediction tasks show significant performance improvements. We also conducted an online A/B test on an online retriever to demonstrate SynAlign's effectiveness.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 35 of its 55 references that are in the map. First posted on arXiv 9 Feb 2025.

Builds on

35 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey14 Jun

Other references · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  8. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  9. 2023Large Language Modelscited by 780
  10. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  11. 2022Large Language Models Can Self-Improvecited by 664
  12. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
22 more
  1. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  2. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 187
  3. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  4. 2023Uncovering ChatGPT's Capabilities in Recommender Systemscited by 161
  5. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  6. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 108
  7. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 94
  8. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  9. 2023Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimescited by 52
  10. 2020GenAug: Data Augmentation for Finetuning Text Generatorscited by 41
  11. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  12. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  13. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  14. 2024TnT-LLM: Text Mining at Scale with Large Language Modelscited by 26
  15. 2023Mixture of Soft Prompts for Controllable Data Generationcited by 23
  16. 2024Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integrationcited by 17
  17. 2018Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actionscited by 16
  18. 2024ERASE: Benchmarking Feature Selection Methods for Deep Recommender Systemscited by 9
  19. 2023Long-Tailed Question Answering in an Open Worldcited by 7
  20. 2024Retrievable Domain-Sensitive Feature Memory for Multi-Domain Recommendationcited by 4
  21. 2025SampleLLM: Optimizing Tabular Data Synthesis in Recommendationscited by 4
  22. 2022A Semi-Synthetic Dataset Generation Framework for Causal Inference in Recommender Systemscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2025Population-Aligned Persona Generation for LLM-based Social Simulationcited by 21
  2. 2025Ensembling LLM-Induced Decision Trees for Explainable and Robust Error Detectioncited by 2
  3. 2025Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluationcited by 1
  4. 2026TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Modelscited by 1
  5. 2026Evaluating Zero-Shot and One-Shot Adaptation of Small Language Models in Leader-Follower Interactioncited by 0
  6. 2026When Hard Negatives Hurt: Bridging the Generative-Discriminative Gap in Hard Negative Synthesis for Retrievalcited by 0
  7. 2026Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesiscited by 0
  8. 2026Carefully Considering Culture: Analyzing LLM Alignment in Single- and Multi-Cultural Settings using Cultural Consensus Theorycited by 0
  9. 2026Behaviorally Grounded User Profiles from the Wild for Personalized Alignment and Multi-Perspective Reasoningcited by 0
  10. 2026Coverage, Not Difficulty, Sets How Much Synthetic Data an Activation Probe Needscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification
  2. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Models
  3. 2025What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning
  4. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
  5. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  6. 2024Efficacy of Synthetic Data as a Benchmark
  7. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  8. 2024Hybrid Training Approaches for LLMs: Leveraging Real and Synthetic Data to Enhance Model Performance in Domain-Specific Applications