# Can I trust my fake data -- A comprehensive quality assessment framework for synthetic tabular data in healthcare

arXiv: https://arxiv.org/abs/2401.13716 · HTML: https://mlsanity.com/paper/2401.13716/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 138 references that are in the map.
- First posted on arXiv 24 Jan 2024.

## Builds on (most cited first, 24 of 24)

- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [Are GANs Created Equal? A Large-Scale Study](https://mlsanity.com/paper/1711.10337/) (2017)
- [How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models](https://mlsanity.com/paper/2102.08921/) (2021)
- [Differentially Private Synthetic Medical Data Generation using Convolutional GANs](https://mlsanity.com/paper/2012.11774/) (2020)
- [Generating Synthetic Mixed-type Longitudinal Electronic Health Records for Artificial Intelligent Applications](https://mlsanity.com/paper/2112.12047/) (2021)
- [A Multifaceted Benchmarking of Synthetic Electronic Health Record Generation Models](https://mlsanity.com/paper/2208.01230/) (2022)
- [Counting Carbon: A Survey of Factors Influencing the Emissions of Machine Learning](https://mlsanity.com/paper/2302.08476/) (2023)
- [CorGAN: Correlation-Capturing Convolutional Generative Adversarial Networks for Generating Synthetic Healthcare Records](https://mlsanity.com/paper/2001.09346/) (2020)
- [Beyond Privacy: Navigating the Opportunities and Challenges of Synthetic Data](https://mlsanity.com/paper/2304.03722/) (2023)
- [Synthetic data, real errors: how (not) to publish and use synthetic data](https://mlsanity.com/paper/2305.09235/) (2023)
- [TabSynDex: A Universal Metric for Robust Evaluation of Synthetic Tabular Data](https://mlsanity.com/paper/2207.05295/) (2022)
- [Longitudinal Variational Autoencoder](https://mlsanity.com/paper/2006.09763/) (2020)
- [Generating Electronic Health Records with Multiple Data Types and Constraints](https://mlsanity.com/paper/2003.07904/) (2020)
- [The Health Gym: Synthetic Health-Related Datasets for the Development of Reinforcement Learning Algorithms](https://mlsanity.com/paper/2203.06369/) (2022)
- [SurvivalGAN: Generating Time-to-Event Data for Survival Analysis](https://mlsanity.com/paper/2302.12749/) (2023)
- [Multi-Label Clinical Time-Series Generation via Conditional GAN](https://mlsanity.com/paper/2204.04797/) (2022)
- [Transitioning from Real to Synthetic data: Quantifying the bias in model](https://mlsanity.com/paper/2105.04144/) (2021)
- [Generating Synthetic Clinical Data that Capture Class Imbalanced Distributions with Generative Adversarial Networks: Example using Antiretroviral Therapy for HIV](https://mlsanity.com/paper/2208.08655/) (2022)
- [Really Useful Synthetic Data -- A Framework to Evaluate the Quality of Differentially Private Synthetic Data](https://mlsanity.com/paper/2004.07740/) (2020)
- [On Utility and Privacy in Synthetic Genomic Data](https://mlsanity.com/paper/2102.03314/) (2021)
- [Deep generative models in DataSHIELD](https://mlsanity.com/paper/2003.07775/) (2020)
- [Synthetic data enable experiments in atomistic machine learning](https://mlsanity.com/paper/2211.16443/) (2022)
- [Precision-Recall Curves Using Information Divergence Frontiers](https://mlsanity.com/paper/1905.10768/) (2019)
- [Predicting Visit Cost of Obstructive Sleep Apnea using Electronic Healthcare Records with Transformer](https://mlsanity.com/paper/2301.12289/) (2023)

## Built on by (most cited first, 17 of 17)

- [Synthetic Data Privacy Metrics](https://mlsanity.com/paper/2501.03941/) (2025)
- [A Consensus Privacy Metrics Framework for Synthetic Data](https://mlsanity.com/paper/2503.04980/) (2025)
- [Ensembling Membership Inference Attacks Against Tabular Generative Models](https://mlsanity.com/paper/2509.05350/) (2025)
- [A Design Framework for operationalizing Trustworthy Artificial Intelligence in Healthcare: Requirements, Tradeoffs and Challenges for its Clinical Adoption](https://mlsanity.com/paper/2504.19179/) (2025)
- [When Tables Leak: Attacking String Memorization in LLM-Based Tabular Data Generation](https://mlsanity.com/paper/2512.08875/) (2025)
- [SHAP Distance: An Explainability-Aware Metric for Evaluating the Semantic Fidelity of Synthetic Tabular Data](https://mlsanity.com/paper/2511.17590/) (2025)
- [SAGE: Sparse Adaptive Guidance for Dependency-Aware Tabular Data Generation](https://mlsanity.com/paper/2604.24368/) (2026)
- [Rethinking Data Augmentation under Covariate Shift: Invariant-Guided Diffusion and Prototype Reweighting](https://mlsanity.com/paper/2610.00873/) (2026)
- [Permissioned Blockchain-based Framework for Ranking Synthetic Data Generators](https://mlsanity.com/paper/2405.07196/) (2024)
- [Opinion: Revisiting synthetic data classifications from a privacy perspective](https://mlsanity.com/paper/2503.03506/) (2025)
- [A Statistical Approach for Synthetic EEG Data Generation](https://mlsanity.com/paper/2504.16143/) (2025)
- [Generating Reliable Synthetic Clinical Trial Data: The Role of Hyperparameter Optimization and Domain Constraints](https://mlsanity.com/paper/2505.05019/) (2025)
- [Synthetic medical data generation: state of the art and application to trauma mechanism classification](https://mlsanity.com/paper/2508.02771/) (2025)
- [Evaluation metrics for temporal preservation in synthetic longitudinal patient data](https://mlsanity.com/paper/2602.10643/) (2026)
- [Evaluating quality in synthetic data generation for large tabular health datasets](https://mlsanity.com/paper/2604.15961/) (2026)
- [Synthetic but Not Realistic: The Evaluation Challenge in Generative Modelling for Structured Electronic Medical Records](https://mlsanity.com/paper/2606.08903/) (2026)
- [FairDiffuseVQVAE: Sampling-Time Fairness in Tabular Diffusion via Conditional Refinement of Vector-Quantized Latents](https://mlsanity.com/paper/2607.28945/) (2026)

## Abstract (arXiv)

Ensuring safe adoption of AI tools in healthcare hinges on access to sufficient data for training, testing and validation. In response to privacy concerns and regulatory requirements, using synthetic data has been suggested. Synthetic data is created by training a generator on real data to produce a dataset with similar statistical properties. Competing metrics with differing taxonomies for quality evaluation have been suggested, resulting in a complex landscape. Optimising quality entails balancing considerations that make the data fit for use, yet relevant dimensions are left out of existing frameworks. We performed a comprehensive literature review on the use of quality evaluation metrics on SD within the scope of tabular healthcare data and SD made using deep generative methods. Based on this and the collective team experiences, we developed a conceptual framework for quality assurance. The applicability was benchmarked against a practical case from the Dutch National Cancer Registry. We present a conceptual framework for quality assurance of SD for AI applications in healthcare that aligns diverging taxonomies, expands on common quality dimensions to include the dimensions of Fairness and Carbon footprint, and proposes stages necessary to support real-life applications. Building trust in synthetic data by increasing transparency and reducing the safety risk will accelerate the development and uptake of trustworthy AI tools for the benefit of patients. Despite the growing emphasis on algorithmic fairness and carbon footprint, these metrics were scarce in the literature review. The overwhelming focus was on statistical similarity using distance metrics while sequential logic detection was scarce. A consensus-backed framework that includes all relevant quality dimensions can provide assurance for safe and responsible real-life applications of SD.