mlsanityData to 7 Oct 2026

A Little Human Data Goes A Long Way

arXiv 2410.13098 ↗First posted 17 Oct 2024cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Faced with an expensive human annotation process, creators of NLP systems increasingly turn to synthetic data generation. While this method shows promise, the extent to which synthetic data can replace human annotation is poorly understood. We investigate the use of synthetic data in Fact Verification (FV) and Question Answering (QA) by studying the effects of incrementally replacing human generated data with synthetic points on eight diverse datasets. Strikingly, replacing up to 90% of the training data only marginally decreases performance, but replacing the final 10% leads to severe declines. We find that models trained on purely synthetic data can be reliably improved by including as few as 125 human generated data points. We show that matching the performance gain of just a little additional human data (only 200 points) requires an order of magnitude more synthetic data and estimate price ratios at which human annotation would be a more cost-effective solution. Our results suggest that even when human annotation at scale is infeasible, there is great value to having a small proportion of the dataset being human generated.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 47 of its 78 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

47 of 78 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  10. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  11. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  12. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
35 more
  1. 2018CoQA: A Conversational Question Answering Challengecited by 674
  2. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  3. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  4. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  5. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  6. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  7. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  8. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  9. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  10. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  11. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  12. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
  13. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  14. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  15. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  16. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  17. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  18. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 103
  19. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 97
  20. 2019Reasoning Over Paragraph Effects in Situationscited by 79
  21. 2022Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehensioncited by 79
  22. 2017Data Noising as Smoothing in Neural Network Language Modelscited by 78
  23. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  24. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapsecited by 75
  25. 2018SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translationcited by 72
  26. 2024Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Imagescited by 41
  27. 2024Can LLMs Replace Manual Annotation of Software Engineering Artifacts?cited by 40
  28. 2024AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotatorscited by 28
  29. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  30. 2024SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluationcited by 23
  31. 2021QAConv: Question Answering on Informative Conversationscited by 17
  32. 2024SynthDST: Synthetic Data is All You Need for Few-Shot Dialog State Trackingcited by 17
  33. 2024A synthetic data approach for domain generalization of NLI modelscited by 15
  34. 2024GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Constructioncited by 15
  35. 2023SciFix: Outperforming GPT3 on Scientific Factual Error Correctioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025A Personalized Conversational Benchmark: Towards Simulating Personalized Conversationscited by 16
  2. 2025When Incentives Backfire, Data Stops Being Humancited by 4
  3. 2025We Need Improved Data Curation and Attribution in AI for Scientific Discoverycited by 1
  4. 2025Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusionscited by 1
  5. 2025Approximating Human Preferences Using a Multi-Judge Learned Systemcited by 0
  6. 2026Toward Interactive Understanding of Code APIscited by 0