mlsanityData to 7 Oct 2026

Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classification

arXiv 2410.21526 ↗First posted 28 Oct 2024cs.LG cs.CLcited by 6

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Synthetic data augmentation via large language models (LLMs) allows researchers to leverage additional training data, thus enhancing the performance of downstream tasks, especially when real-world data is scarce. However, the generated data can deviate from the real-world data, and this misalignment can bring deficient outcomes while applying the trained model to applications. Therefore, we proposed efficient weighted-loss approaches to align synthetic data with real-world distribution by emphasizing high-quality and diversified data generated by LLMs with using merely a little real-world data. We empirically assessed the effectiveness of our method on multiple text classification tasks, and the results showed leveraging our approaches on a BERT-level model robustly outperformed standard cross-entropy and other data weighting approaches, providing potential solutions to effectively leveraging synthetic data from any suitable data generator for model training.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 26 of its 41 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

26 of 41 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  6. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  7. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  8. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
  9. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  10. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  11. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  12. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
14 more
  1. 2018What is the Effect of Importance Weighting in Deep Learning?cited by 250
  2. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  3. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  4. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  5. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
  6. 2015Online Batch Selection for Faster Training of Neural Networkscited by 121
  7. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 96
  8. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  9. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  10. 2019Learning Data Manipulation for Augmentation and Weightingcited by 44
  11. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  12. 2020What Makes a "Good" Data Augmentation in Knowledge Distillation -- A Statistical Perspectivecited by 35
  13. 2023Towards Accelerated Model Training via Bayesian Data Selectioncited by 11
  14. 2024UniGen: Universal Domain Generalization for Sentiment Classification via Zero-shot Dataset Generationcited by 3

Built on by

Cited by year

Citing papers per year: 2025: 6.

6 papers.

  1. 2025DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Datacited by 6
  2. 2025Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Themcited by 4
  3. 2025Feeding LLM Annotations to BERT Classifiers at Your Own Riskcited by 2
  4. 2025What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuningcited by 1
  5. 2025SynQuE: Estimating Synthetic Dataset Quality Without Annotationscited by 1
  6. 2025A Novel Data Augmentation Approach for Automatic Speaking Assessment on Opinion Expressionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficacy of Synthetic Data as a Benchmark
  2. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  3. 2023Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models
  4. 2025Augmenting Human-Annotated Training Data with Large Language Model Generation and Distillation in Open-Response Assessment
  5. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification
  6. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  7. 2025Scaling Laws of Synthetic Data for Language Models
  8. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective