# Efficacy of Synthetic Data as a Benchmark

arXiv: https://arxiv.org/abs/2409.11968 · HTML: https://mlsanity.com/paper/2409.11968/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 10 of 10)

- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [Does Synthetic Data Generation of LLMs Help Clinical Text Mining?](https://mlsanity.com/paper/2303.04360/) (2023)
- [From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models](https://mlsanity.com/paper/2212.10846/) (2022)
- [CrossNER: Evaluating Cross-Domain Named Entity Recognition](https://mlsanity.com/paper/2012.04373/) (2020)
- [Secure and Robust Machine Learning for Healthcare: A Survey](https://mlsanity.com/paper/2001.08103/) (2020)
- [FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering](https://mlsanity.com/paper/2308.12060/) (2023)
- [Deflating Dataset Bias Using Synthetic Data Augmentation](https://mlsanity.com/paper/2004.13866/) (2020)
- [How much real data do we actually need: Analyzing object detection performance using synthetic and real data](https://mlsanity.com/paper/1907.07061/) (2019)

## Built on by (most cited first, 30 of 30)

- [VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Models](https://mlsanity.com/paper/2411.09105/) (2024)
- [Understanding and Mitigating Bias Inheritance in LLM-based Data Augmentation on Downstream Tasks](https://mlsanity.com/paper/2502.04419/) (2025)
- [What Has Been Lost with Synthetic Evaluation?](https://mlsanity.com/paper/2505.22830/) (2025)
- [DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models](https://mlsanity.com/paper/2601.11895/) (2026)
- [LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient](https://mlsanity.com/paper/2502.01683/) (2025)
- [RECAP: REwriting Conversations for Intent Understanding in Agentic Planning](https://mlsanity.com/paper/2509.04472/) (2025)
- [AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance](https://mlsanity.com/paper/2510.01474/) (2025)
- [Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator](https://mlsanity.com/paper/2505.20738/) (2025)
- [Virus Infection Attack on LLMs: Your Poisoning Can Spread "VIA" Synthetic Data](https://mlsanity.com/paper/2509.23041/) (2025)
- [Breaking the Silence: A Dataset and Benchmark for Bangla Text-to-Gloss Translation](https://mlsanity.com/paper/2504.02293/) (2025)
- [When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation](https://mlsanity.com/paper/2509.26600/) (2025)
- [Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe](https://mlsanity.com/paper/2601.08847/) (2025)
- [Artificial Conversations, Real Results: Fostering Language Detection with Synthetic Data](https://mlsanity.com/paper/2503.24062/) (2025)
- [Modelling and Classifying the Components of a Literature Review](https://mlsanity.com/paper/2508.04337/) (2025)
- [Generative Models for Synthetic Data: Transforming Data Mining in the GenAI Era](https://mlsanity.com/paper/2508.19570/) (2025)
- [How Well Do Vision--Language Models Understand Cities? A Comparative Study on Spatial Reasoning from Street-View Images](https://mlsanity.com/paper/2508.21565/) (2025)
- [LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs](https://mlsanity.com/paper/2511.14774/) (2025)
- [SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations](https://mlsanity.com/paper/2605.22564/) (2026)
- [SkillScope: A Tool to Predict Fine-Grained Skills Needed to Solve Issues on GitHub](https://mlsanity.com/paper/2501.15922/) (2025)
- [Evaluating LLMs for Demographic-Targeted Social Bias Detection: A Comprehensive Benchmark Study](https://mlsanity.com/paper/2510.04641/) (2025)
- [Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents](https://mlsanity.com/paper/2608.06329/) (2026)
- [DevNous: An LLM-Based Multi-Agent System for Grounding IT Project Management in Unstructured Conversation](https://mlsanity.com/paper/2508.08761/) (2025)
- [LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.06198/) (2026)
- [SeaAlert: Robust Severity Classification and LLM-Based Information Extraction for Noisy Maritime Distress Communications](https://mlsanity.com/paper/2604.14163/) (2026)
- [Synthetic Data in Education: Empirical Insights from Traditional Resampling and Deep Generative Models](https://mlsanity.com/paper/2604.21031/) (2026)
- [On Privacy in Data-Space Tabular Diffusion Models: Influential Factors, Attacker Knowledge, and Metrics](https://mlsanity.com/paper/2605.06835/) (2026)
- [NodeSynth: Socially Aligned Synthetic Data for AI Evaluation](https://mlsanity.com/paper/2605.14381/) (2026)
- [PromptResponse: Optimizing Prompts for LLM Coding Tasks](https://mlsanity.com/paper/2608.21074/) (2026)
- [Incremental Risk Assessment of Progressive Elder Financial Scams via Instruction-Tuned Small Language Models](https://mlsanity.com/paper/2609.00005/) (2026)
- [Adapting Vision-Language Models for Human-Readable XAI in Industrial Object Detection](https://mlsanity.com/paper/2609.31690/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have enabled a range of applications in zero-shot and few-shot learning settings, including the generation of synthetic datasets for training and testing. However, to reliably use these synthetic datasets, it is essential to understand how representative they are of real-world data. We investigate this by assessing the effectiveness of generating synthetic data through LLM and using it as a benchmark for various NLP tasks. Our experiments across six datasets, and three different tasks, show that while synthetic data can effectively capture performance of various methods for simpler tasks, such as intent classification, it falls short for more complex tasks like named entity recognition. Additionally, we propose a new metric called the bias factor, which evaluates the biases introduced when the same LLM is used to both generate benchmarking data and to perform the tasks. We find that smaller LLMs exhibit biases towards their own generated data, whereas larger models do not. Overall, our findings suggest that the effectiveness of synthetic data as a benchmark varies depending on the task, and that practitioners should rely on data generated from multiple larger models whenever possible.