mlsanityData to 14 Oct 2026

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs

arXiv 2503.12347 ↗First posted 16 Mar 2025cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Synthetic data offers a promising path to train models while preserving data privacy. Differentially private (DP) finetuning of large language models (LLMs) as data generator is effective, but is impractical when computation resources are limited. Meanwhile, prompt-based methods such as private evolution depend heavily on the manual prompts, and ineffectively use private information in their iterative data selection process. To overcome these limitations, we propose CTCL (Data Synthesis with ConTrollability and CLustering), a novel framework for generating privacy-preserving synthetic data without extensive prompt engineering or billion-scale LLM finetuning. CTCL pretrains a lightweight 140M conditional generator and a clustering-based topic model on large-scale public data. To further adapt to the private domain, the generator is DP finetuned on private data for fine-grained textual information, while the topic model extracts a DP histogram representing distributional information. The DP generator then samples according to the DP histogram to synthesize a desired number of data examples. Evaluation across five diverse domains demonstrates the effectiveness of our framework, particularly in the strong privacy regime. Systematic ablation validates the design of each framework component and highlights the scalability of our approach.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 46 of its 65 references that are in the map. First posted on arXiv 16 Mar 2025.

Builds on

46 of 65 references.

References · 46

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2016Deep Learning with Differential Privacycited by 2,728
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  10. 2023Extracting Training Data from Diffusion Modelscited by 732
  11. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  12. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
34 more
  1. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  2. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  3. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  4. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 267
  5. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  6. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  7. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  8. 2019Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbationscited by 119
  9. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  10. 2014Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Boundscited by 107
  11. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  12. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 83
  13. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 76
  14. 2022Differentially Private Language Models for Secure Data Sharingcited by 68
  15. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 66
  16. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 65
  17. 2023Harnessing large-language models to generate private synthetic textcited by 65
  18. 2022The Limits of Word Level Differential Privacycited by 63
  19. 2023Locally Differentially Private Document Generation Using Zero Shot Promptingcited by 63
  20. 2023Privacy-Preserving In-Context Learning for Large Language Modelscited by 50
  21. 2021TEM: High Utility Metric Differential Privacy on Textcited by 43
  22. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35
  23. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  24. 2024Private prediction for large-scale synthetic text generationcited by 24
  25. 2025Metadata Conditioning Accelerates Language Model Pre-trainingcited by 22
  26. 2024Prompt Public Large Language Models to Synthesize Data for Private On-device Applicationscited by 17
  27. 2024Federated Learning in Practice: Reflections and Projectionscited by 15
  28. 2024KnowledgeSG: Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Servercited by 11
  29. 2025Contrastive Private Data Synthesis via Weighted Multi-PLM Fusioncited by 11
  30. 2023Synthetic Query Generation for Privacy-Preserving Deep Retrieval Systems using Differentially Private Language Modelscited by 10
  31. 2024Private Synthetic Text Generation with Diffusion Modelscited by 10
  32. 2023RedCoast: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUscited by 3
  33. 2024SoftSRV: Learn to Generate Targeted Synthetic Datacited by 3
  34. 2025Private Text Generation by Seeding Large Language Model Promptscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2025Struct-Bench: A Benchmark for Differentially Private Structured Text Generationcited by 7
  2. 2025Clustering and Median Aggregation Improve Differentially Private Inferencecited by 5
  3. 2025ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Controlcited by 3
  4. 2025Synthesizing and Adapting Error Correction Data for Mobile Large Language Model Applicationscited by 2
  5. 2026CoLA: A Choice Leakage Attack Framework to Expose Privacy Risks in Subset Trainingcited by 2
  6. 2025LLMSynthor: Macro-Aligned Micro-Records Synthesis with Large Language Modelscited by 1
  7. 2026Differentially Private Preference Data Synthesis for Large Language Model Alignmentcited by 1
  8. 2025Private Training & Data Generation by Clustering Embeddingscited by 0
  9. 2025Secret-Protected Evolution for Differentially Private Synthetic Text Generationcited by 0
  10. 2026DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuningcited by 0
  11. 2026MAPLE: Metadata Augmented Private Language Evolutioncited by 0
  12. 2026Model in Distress: Sentiment Analysis on French Synthetic Social Mediacited by 0
4 more
  1. 2026ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?cited by 0
  2. 2026The Privacy-Hallucination Tradeoff in Differentially Private Language Modelscited by 0
  3. 2026Only Pay What You Must Spend: On-Demand Privacy Budget Payment for Differentially Private RAGcited by 0
  4. 2026Tight End-to-End Accounting for Differentially Private Synthetic Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SafeSynthDP: Leveraging Large Language Models for Privacy-Preserving Synthetic Data Generation Using Differential Privacy
  2. 2025SynBench: A Benchmark for Differentially Private Text Generation
  3. 2026Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generation
  4. 2024Differentially Private Tabular Data Synthesis using Large Language Models
  5. 2026Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation
  6. 2024Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data
  7. 2024Differentially Private Learning Needs Better Model Initialization and Self-Distillation
  8. 2024Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domains