# Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domains

arXiv: https://arxiv.org/abs/2410.08327 · HTML: https://mlsanity.com/paper/2410.08327/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Privacy-preserving tabular data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 28 of its 53 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [Privacy-preserving tabular data synthesis](https://mlsanity.com/synth/privacy-preserving-tabular-data-synthesis/)

## Builds on (most cited first, 28 of 28)

- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Numerical Composition of Differential Privacy](https://mlsanity.com/paper/2106.02848/) (2021)
- [How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence](https://mlsanity.com/paper/2004.12158/) (2020)
- [Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbations](https://mlsanity.com/paper/1910.08902/) (2019)
- [Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe](https://mlsanity.com/paper/2210.14348/) (2022)
- [Differential Privacy for Text Analytics via Natural Text Sanitization](https://mlsanity.com/paper/2106.01221/) (2021)
- [Differentially Private Language Models for Secure Data Sharing](https://mlsanity.com/paper/2210.13918/) (2022)
- [Harnessing large-language models to generate private synthetic text](https://mlsanity.com/paper/2306.01684/) (2023)
- [The Limits of Word Level Differential Privacy](https://mlsanity.com/paper/2205.02130/) (2022)
- [Selective Differential Privacy for Language Modeling](https://mlsanity.com/paper/2108.12944/) (2021)
- [A Customized Text Sanitization Mechanism with Differential Privacy](https://mlsanity.com/paper/2207.01193/) (2022)
- [Coreference Resolution without Span Representations](https://mlsanity.com/paper/2101.00434/) (2021)
- [Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference](https://mlsanity.com/paper/2011.10819/) (2020)
- [Fine-Tuning Large Language Models with User-Level Differential Privacy](https://mlsanity.com/paper/2407.07737/) (2024)
- [Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning](https://mlsanity.com/paper/2406.14322/) (2024)
- [The Limits of Differential Privacy (and its Misuse in Data Release and Machine Learning)](https://mlsanity.com/paper/2011.02352/) (2020)
- [Privacy- and Utility-Preserving NLP with Anonymized Data: A case study of Pseudonymization](https://mlsanity.com/paper/2306.05561/) (2023)
- [An Empirical Evaluation of Deep Learning for ICD-9 Code Assignment using MIMIC-III Clinical Notes](https://mlsanity.com/paper/1802.02311/) (2018)
- [Generation of Synthetic Electronic Medical Record Text](https://mlsanity.com/paper/1812.02793/) (2018)
- [Mention Annotations Alone Enable Efficient Domain Adaptation for Coreference Resolution](https://mlsanity.com/paper/2210.07602/) (2022)
- [For Generated Text, Is NLI-Neutral Text the Best Text?](https://mlsanity.com/paper/2302.08577/) (2023)

## Built on by (most cited first, 12 of 12)

- [ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoning](https://mlsanity.com/paper/2502.14860/) (2025)
- [A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage](https://mlsanity.com/paper/2504.21035/) (2025)
- [SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domains](https://mlsanity.com/paper/2507.07229/) (2025)
- [Struct-Bench: A Benchmark for Differentially Private Structured Text Generation](https://mlsanity.com/paper/2509.10696/) (2025)
- [How do we measure privacy in text? A survey of text anonymization metrics](https://mlsanity.com/paper/2512.01109/) (2025)
- [Controlled Generation for Private Synthetic Text](https://mlsanity.com/paper/2509.25729/) (2025)
- [Privacy-Preserving Generation of Clinical Narratives from Medical Terminologies](https://mlsanity.com/paper/2509.10882/) (2025)
- [Leveraging Pretrained Language Models as Energy Functions for Glauber Dynamics Text Diffusion](https://mlsanity.com/paper/2605.04291/) (2026)
- [Measuring Privacy Risks and Tradeoffs in Financial Synthetic Data Generation](https://mlsanity.com/paper/2602.09288/) (2026)
- [The Conundrum of Trustworthy Research on Attacking Personally Identifiable Information Removal Techniques](https://mlsanity.com/paper/2603.08207/) (2026)
- [ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?](https://mlsanity.com/paper/2606.01849/) (2026)
- [The Privacy-Hallucination Tradeoff in Differentially Private Language Models](https://mlsanity.com/paper/2609.00492/) (2026)

## Abstract (arXiv)

The difficulty of anonymizing text data hinders the development and deployment of NLP in high-stakes domains that involve private data, such as healthcare and social services. Poorly anonymized sensitive data cannot be easily shared with annotators or external researchers, nor can it be used to train public models. In this work, we explore the feasibility of using synthetic data generated from differentially private language models in place of real data to facilitate the development of NLP in these domains without compromising privacy. In contrast to prior work, we generate synthetic data for real high-stakes domains, and we propose and conduct use-inspired evaluations to assess data quality. Our results show that prior simplistic evaluations have failed to highlight utility, privacy, and fairness issues in the synthetic data. Overall, our work underscores the need for further improvements to synthetic data generation for it to be a viable way to enable privacy-preserving data sharing.