# A density ratio framework for evaluating the utility of synthetic data

arXiv: https://arxiv.org/abs/2408.13167 · HTML: https://mlsanity.com/paper/2408.13167/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 61 references that are in the map.
- First posted on arXiv 23 Aug 2024.

## Builds on (most cited first, 8 of 8)

- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [Learning in Implicit Generative Models](https://mlsanity.com/paper/1610.03483/) (2016)
- [Classification accuracy as a proxy for two sample testing](https://mlsanity.com/paper/1602.02210/) (2016)
- [Featurized Density Ratio Estimation](https://mlsanity.com/paper/2107.02212/) (2021)
- [Generative Adversarial Nets from a Density Ratio Estimation Perspective](https://mlsanity.com/paper/1610.02920/) (2016)
- [High-Dimensional Density Ratio Estimation with Extensions to Approximate Likelihood Computation](https://mlsanity.com/paper/1404.7063/) (2014)
- [Mitigating Statistical Bias within Differentially Private Synthetic Data](https://mlsanity.com/paper/2108.10934/) (2021)
- [Think before you shrink: Alternatives to default shrinkage methods can improve prediction accuracy, calibration and coverage](https://mlsanity.com/paper/2301.09890/) (2023)

## Abstract (arXiv)

Synthetic data generation is a promising technique to facilitate the use of sensitive data while mitigating the risk of privacy breaches. However, for synthetic data to be useful in downstream analysis tasks, it needs to be of sufficient quality. Various methods have been proposed to measure the utility of synthetic data, but their results are often incomplete or even misleading. In this paper, we propose using density ratio estimation to improve quality evaluation for synthetic data, and thereby the quality of synthesized datasets. We show how this framework relates to and builds on existing measures, yielding global and local utility measures that are informative and easy to interpret. We develop an estimator which requires little to no manual tuning due to automatic selection of a nonparametric density ratio model. Through simulations, we find that density ratio estimation yields more accurate estimates of global utility than established procedures. A real-world data application demonstrates how the density ratio can guide refinements of synthesis models and can be used to improve downstream analyses. We conclude that density ratio estimation is a valuable tool in synthetic data generation workflows and provide these methods in the accessible open source R-package densityratio.