mlsanityData to 7 Oct 2026

Harnessing large-language models to generate private synthetic text

arXiv 2306.01684 ↗First posted 2 Jun 2023cs.LG cs.CRcited by 65

Not on the map. Why?

Abstract (arXiv)

Differentially private training algorithms like DP-SGD protect sensitive training data by ensuring that trained models do not reveal private information. An alternative approach, which this paper studies, is to use a sensitive dataset to generate synthetic data that is differentially private with respect to the original data, and then non-privately training a model on the synthetic data. Doing so has several advantages: synthetic data can be reused for other tasks (including for hyper parameter tuning), retained indefinitely, and shared with third parties without sacrificing privacy. However, generating private synthetic data is much harder than training a private model. To improve performance on text data, recent work has utilized public data by starting with a pre-trained generative language model and privately fine-tuning it on sensitive data. This model can be used to sample a DP synthetic dataset. While this strategy seems straightforward, executing it has proven problematic. Previous approaches either show significant performance loss, or have, as we show, critical design flaws. In this paper we demonstrate that a proper training objective along with tuning fewer parameters results in excellent DP synthetic data quality. Our approach is competitive with direct DP-training of downstream classifiers in terms of performance on downstream tasks. Further, we demonstrate that our DP synthetic data is not only useful for downstream classifier training, but also to tune those same models.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 35 of its 56 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

35 of 56 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2016Deep Learning with Differential Privacycited by 2,682
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
23 more
  1. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  2. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  3. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  4. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  5. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  6. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  7. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  8. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  9. 2018Differentially Private Generative Adversarial Networkcited by 235
  10. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  11. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  12. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  13. 2013Fingerprinting Codes and the Price of Approximate Differential Privacycited by 81
  14. 2021Benchmarking Differentially Private Synthetic Data Generation Algorithmscited by 80
  15. 2022Differentially Private Language Models for Secure Data Sharingcited by 68
  16. 2021Iterative Methods for Private Synthetic Data: Unifying Framework and New Methodscited by 58
  17. 2020Differentially Private Synthetic Medical Data Generation using Convolutional GANscited by 52
  18. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  19. 2017Practical Locally Private Heavy Hitterscited by 28
  20. 2020The Trade-Offs of Private Predictioncited by 22
  21. 2019Towards Automatic Generation of Shareable Synthetic Clinical Notes Using Neural Language Modelscited by 20
  22. 2018Generation of Synthetic Electronic Medical Record Textcited by 14
  23. 2022Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corporacited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 16; 2025: 27; 2026: 18.

65 papers.

Sort:

By area:

  1. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  2. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  3. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 74
  4. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 34
  5. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 30
  6. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  7. 2024Fine-Tuning Large Language Models with User-Level Differential Privacycited by 28
  8. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  9. 2024Privacy Preserving Prompt Engineering: A Surveycited by 24
  10. 2024Private prediction for large-scale synthetic text generationcited by 24
  11. 2024Differentially Private Knowledge Distillation via Synthetic Text Generationcited by 18
  12. 2025The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Textcited by 18
38 more
  1. 2024Prompt Public Large Language Models to Synthesize Data for Private On-device Applicationscited by 17
  2. 2024Differentially Private Tabular Data Synthesis using Large Language Modelscited by 16
  3. 2025Language Agents as Digital Representatives in Collective Decision-Makingcited by 15
  4. 2025Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMscited by 15
  5. 2025A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakagecited by 15
  6. 2025POPri: Private Federated Learning using Preference-Optimized Synthetic Datacited by 14
  7. 2024Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domainscited by 12
  8. 2024KnowledgeSG: Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Servercited by 11
  9. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11
  10. 2025InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacycited by 10
  11. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
  12. 2025Synthesize Privacy-Preserving High-Resolution Images via Private Textual Intermediariescited by 8
  13. 2025From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactionscited by 7
  14. 2023Machine Learning needs Better Randomness Standards: Randomised Smoothing and PRNG-based attackscited by 6
  15. 2025Is API Access to LLMs Useful for Generating Private Synthetic Tabular Data?cited by 5
  16. 2025Clustering and Median Aggregation Improve Differentially Private Inferencecited by 5
  17. 2025Differentially Private Synthetic Text Generation for Retrieval-Augmented Generation (RAG)cited by 5
  18. 2025PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillationcited by 4
  19. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  20. 2023Hot PATE: Private Aggregation of Distributions for Diverse Taskcited by 3
  21. 2024Towards Realistic Synthetic User-Generated Content: A Scaffolding Approach to Generating Online Discussionscited by 3
  22. 2025RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesiscited by 3
  23. 2025Differential Privacy: Gradient Leakage Attacks in Federated Learning Environmentscited by 3
  24. 2024Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Datacited by 2
  25. 2025Synthesizing and Adapting Error Correction Data for Mobile Large Language Model Applicationscited by 2
  26. 2025Privacy Preserving In-Context-Learning Framework for Large Language Modelscited by 2
  27. 2025Controlled Generation for Private Synthetic Textcited by 2
  28. 2025ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Controlcited by 2
  29. 2026EPSVec: Efficient and Private Synthetic Data Generation via Dataset Vectorscited by 2
  30. 2026Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generationcited by 2
  31. 2026Differentially Private Synthetic Data via APIs 4: Tabular Datacited by 2
  32. 2023Synthetic Text Generation using Hypergraph Representationscited by 1
  33. 2024Bayesian Power Steering: An Effective Approach for Domain Adaptation of Diffusion Modelscited by 1
  34. 2025Tiny QA Benchmark++: Ultra-Lightweight, Synthetic Multilingual Dataset Generation & Smoke-Tests for Continuous LLM Evaluationcited by 1
  35. 2025Privacy-Preserving Generation of Clinical Narratives from Medical Terminologiescited by 1
  36. 2026Privately Fine-Tuned LLMs Preserve Temporal Dynamics in Tabular Datacited by 1
  37. 2026DP-OPD: Differentially Private On-Policy Distillation for Language Modelscited by 1
  38. 2025Collaborative LLM Numerical Reasoning with Local Data Protectioncited by 0

50 most cited of 65 All 65 (JSON)