# A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models

arXiv: https://arxiv.org/abs/2404.14445 · HTML: https://mlsanity.com/paper/2404.14445/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Privacy-preserving tabular data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 13 of its 37 references that are in the map.
- First posted on arXiv 20 Apr 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [Privacy-preserving tabular data synthesis](https://mlsanity.com/synth/privacy-preserving-tabular-data-synthesis/)

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders](https://mlsanity.com/paper/2403.03952/) (2024)
- [Does Synthetic Data Generation of LLMs Help Clinical Text Mining?](https://mlsanity.com/paper/2303.04360/) (2023)
- [Training generative neural networks via Maximum Mean Discrepancy optimization](https://mlsanity.com/paper/1505.03906/) (2015)
- [Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations](https://mlsanity.com/paper/2310.07849/) (2023)
- [Adversarial Feature Matching for Text Generation](https://mlsanity.com/paper/1706.03850/) (2017)
- [A Tutorial on Learning With Bayesian Networks](https://mlsanity.com/paper/2002.00269/) (2020)
- [CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training](https://mlsanity.com/paper/1703.10155/) (2017)
- [A Comparative Study of Open-Source Large Language Models, GPT-4 and Claude 2: Multiple-Choice Test Taking in Nephrology](https://mlsanity.com/paper/2308.04709/) (2023)

## Built on by (most cited first, 8 of 8)

- [Struct-Bench: A Benchmark for Differentially Private Structured Text Generation](https://mlsanity.com/paper/2509.10696/) (2025)
- [GReaTER: Generate Realistic Tabular data after data Enhancement and Reduction](https://mlsanity.com/paper/2503.15564/) (2025)
- [Can Large Language Models Generate Effective Datasets for Emotion Recognition in Conversations?](https://mlsanity.com/paper/2508.05474/) (2025)
- [Dataset Creation for Visual Entailment using Generative AI](https://mlsanity.com/paper/2508.11605/) (2025)
- [RL-Finetuned LLMs for Privacy-Preserving Synthetic Rewriting](https://mlsanity.com/paper/2508.19286/) (2025)
- [SynQP: A Framework and Metrics for Evaluating the Quality and Privacy Risk of Synthetic Data](https://mlsanity.com/paper/2601.12124/) (2026)
- [NodeSynth: Socially Aligned Synthetic Data for AI Evaluation](https://mlsanity.com/paper/2605.14381/) (2026)
- [Beyond the Stitching Assumption: A Unified Framework for Multimodal Synthetic Data Evaluation via Semantic Quantization](https://mlsanity.com/paper/2609.22149/) (2026)

## Abstract (arXiv)

The rapid advancements in generative AI and large language models (LLMs) have opened up new avenues for producing synthetic data, particularly in the realm of structured tabular formats, such as product reviews. Despite the potential benefits, concerns regarding privacy leakage have surfaced, especially when personal information is utilized in the training datasets. In addition, there is an absence of a comprehensive evaluation framework capable of quantitatively measuring the quality of the generated synthetic data and their utility for downstream tasks. In response to this gap, we introduce SynEval, an open-source evaluation framework designed to assess the fidelity, utility, and privacy preservation of synthetically generated tabular data via a suite of diverse evaluation metrics. We validate the efficacy of our proposed framework - SynEval - by applying it to synthetic product review data generated by three state-of-the-art LLMs: ChatGPT, Claude, and Llama. Our experimental findings illuminate the trade-offs between various evaluation metrics in the context of synthetic data generation. Furthermore, SynEval stands as a critical instrument for researchers and practitioners engaged with synthetic tabular data,, empowering them to judiciously determine the suitability of the generated data for their specific applications, with an emphasis on upholding user privacy.