# Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generation

arXiv: https://arxiv.org/abs/2604.07486 · HTML: https://mlsanity.com/paper/2604.07486/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Privacy-preserving tabular data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 38 of its 59 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Privacy-preserving tabular data synthesis](https://mlsanity.com/synth/privacy-preserving-tabular-data-synthesis/)

## Builds on (most cited first, 30 of 38)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023)
- [Scalable Private Learning with PATE](https://mlsanity.com/paper/1802.08908/) (2018)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT](https://mlsanity.com/paper/2303.04226/) (2023)
- [Does Synthetic Data Generation of LLMs Help Clinical Text Mining?](https://mlsanity.com/paper/2303.04360/) (2023)
- [LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs](https://mlsanity.com/paper/2408.07055/) (2024)
- [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://mlsanity.com/paper/1908.08962/) (2019)
- [Machine Learning for Synthetic Data Generation: A Review](https://mlsanity.com/paper/2302.04062/) (2023)
- [Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe](https://mlsanity.com/paper/2210.14348/) (2022)
- [Differentially Private Synthetic Data via Foundation Model APIs 2: Text](https://mlsanity.com/paper/2403.01749/) (2024)
- [Differentially Private Language Models for Secure Data Sharing](https://mlsanity.com/paper/2210.13918/) (2022)
- [Harnessing large-language models to generate private synthetic text](https://mlsanity.com/paper/2306.01684/) (2023)
- [Jointly Measuring Diversity and Quality in Text Generation Models](https://mlsanity.com/paper/1904.03971/) (2019)
- [Reducing Privacy Risks in Online Self-Disclosures with Language Models](https://mlsanity.com/paper/2311.09538/) (2023)
- [Memorization in NLP Fine-tuning Methods](https://mlsanity.com/paper/2205.12506/) (2022)
- [A Synthetic Dataset for Personal Attribute Inference](https://mlsanity.com/paper/2406.07217/) (2024)

## Built on by (most cited first, 2 of 2)

- [Aggregation-Aware Synthetic Text Generation Against Authorship Re-Identification](https://mlsanity.com/paper/2608.22161/) (2026)
- [The Privacy-Hallucination Tradeoff in Differentially Private Language Models](https://mlsanity.com/paper/2609.00492/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have emerged as a powerful tool for synthetic data generation. A particularly important use case is producing synthetic replicas of private text, which requires carefully balancing privacy and utility. We propose Realistic and Privacy-Preserving Synthetic Data Generation (RPSG), which uses private seeds and integrates privacy-preserving strategies, including a formal differential privacy (DP) mechanism in the candidate selection, to generate realistic synthetic data. Comprehensive experiments against state-of-the-art private synthetic data generation methods demonstrate that RPSG achieves high fidelity to private data while providing strong privacy protection.