mlsanityData to 7 Oct 2026

Efficacy of Synthetic Data as a Benchmark

arXiv 2409.11968 ↗First posted 18 Sep 2024cs.CL cs.LGcited by 29

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Large language models (LLMs) have enabled a range of applications in zero-shot and few-shot learning settings, including the generation of synthetic datasets for training and testing. However, to reliably use these synthetic datasets, it is essential to understand how representative they are of real-world data. We investigate this by assessing the effectiveness of generating synthetic data through LLM and using it as a benchmark for various NLP tasks. Our experiments across six datasets, and three different tasks, show that while synthetic data can effectively capture performance of various methods for simpler tasks, such as intent classification, it falls short for more complex tasks like named entity recognition. Additionally, we propose a new metric called the bias factor, which evaluates the biases introduced when the same LLM is used to both generate benchmarking data and to perform the tasks. We find that smaller LLMs exhibit biases towards their own generated data, whereas larger models do not. Overall, our findings suggest that the effectiveness of synthetic data as a benchmark varies depending on the task, and that practitioners should rely on data generated from multiple larger models whenever possible.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 10 of its 16 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

10 of 16 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey14 Jun

Other references · 9

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  3. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  4. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  5. 2020CrossNER: Evaluating Cross-Domain Named Entity Recognitioncited by 102
  6. 2020Secure and Robust Machine Learning for Healthcare: A Surveycited by 77
  7. 2023FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answeringcited by 51
  8. 2020Deflating Dataset Bias Using Synthetic Data Augmentationcited by 37
  9. 2019How much real data do we actually need: Analyzing object detection performance using synthetic and real datacited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 18; 2026: 10.

29 papers.

Sort:

By area:

  1. 2024VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Modelscited by 10
  2. 2025Understanding and Mitigating Bias Inheritance in LLM-based Data Augmentation on Downstream Taskscited by 9
  3. 2025What Has Been Lost with Synthetic Evaluation?cited by 8
  4. 2026DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Modelscited by 8
  5. 2025RECAP: REwriting Conversations for Intent Understanding in Agentic Planningcited by 7
  6. 2025LLM-Powered Benchmark Factory: Reliable, Generic, and Efficientcited by 6
  7. 2025AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliancecited by 5
  8. 2025Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generatorcited by 4
  9. 2025Breaking the Silence: A Dataset and Benchmark for Bangla Text-to-Gloss Translationcited by 3
  10. 2025Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universecited by 3
  11. 2025Artificial Conversations, Real Results: Fostering Language Detection with Synthetic Datacited by 2
  12. 2025Modelling and Classifying the Components of a Literature Reviewcited by 2
17 more
  1. 2025Generative Models for Synthetic Data: Transforming Data Mining in the GenAI Eracited by 2
  2. 2025Virus Infection Attack on LLMs: Your Poisoning Can Spread "VIA" Synthetic Datacited by 2
  3. 2025When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluationcited by 2
  4. 2025LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMscited by 2
  5. 2025SkillScope: A Tool to Predict Fine-Grained Skills Needed to Solve Issues on GitHubcited by 1
  6. 2025How Well Do Vision--Language Models Understand Cities? A Comparative Study on Spatial Reasoning from Street-View Imagescited by 1
  7. 2025Evaluating LLMs for Demographic-Targeted Social Bias Detection: A Comprehensive Benchmark Studycited by 1
  8. 2026SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluationscited by 1
  9. 2026Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agentscited by 1
  10. 2025DevNous: An LLM-Based Multi-Agent System for Grounding IT Project Management in Unstructured Conversationcited by 0
  11. 2026LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generationcited by 0
  12. 2026SeaAlert: Robust Severity Classification and LLM-Based Information Extraction for Noisy Maritime Distress Communicationscited by 0
  13. 2026Synthetic Data in Education: Empirical Insights from Traditional Resampling and Deep Generative Modelscited by 0
  14. 2026On Privacy in Data-Space Tabular Diffusion Models: Influential Factors, Attacker Knowledge, and Metricscited by 0
  15. 2026NodeSynth: Socially Aligned Synthetic Data for AI Evaluationcited by 0
  16. 2026PromptResponse: Optimizing Prompts for LLM Coding Taskscited by 0
  17. 2026Incremental Risk Assessment of Progressive Elder Financial Scams via Instruction-Tuned Small Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  2. 2025Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
  3. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  4. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Models
  5. 2025Scaling Laws of Synthetic Data for Language Models
  6. 2025Does Training on Synthetic Data Make Models Less Robust?
  7. 2024Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models
  8. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data