# PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs

arXiv: https://arxiv.org/abs/2406.02958 · HTML: https://mlsanity.com/paper/2406.02958/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 102 of its 130 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 30 of 102)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Scalable Private Learning with PATE](https://mlsanity.com/paper/1802.08908/) (2018)
- [Differentially Private Learning Needs Better Features (or Much More Data)](https://mlsanity.com/paper/2011.11660/) (2020)
- [Unlearn What You Want to Forget: Efficient Unlearning for LLMs](https://mlsanity.com/paper/2310.20150/) (2023)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)

## Built on by (most cited first, 29 of 29)

- [Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs](https://mlsanity.com/paper/2503.12347/) (2025)
- [POPri: Private Federated Learning using Preference-Optimized Synthetic Data](https://mlsanity.com/paper/2504.16438/) (2025)
- [Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Model](https://mlsanity.com/paper/2502.05505/) (2025)
- [Struct-Bench: A Benchmark for Differentially Private Structured Text Generation](https://mlsanity.com/paper/2509.10696/) (2025)
- [Is API Access to LLMs Useful for Generating Private Synthetic Tabular Data?](https://mlsanity.com/paper/2502.06555/) (2025)
- [FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2506.09199/) (2025)
- [FedSpaLLM: Federated Pruning of Large Language Models](https://mlsanity.com/paper/2410.14852/) (2024)
- [RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis](https://mlsanity.com/paper/2502.18517/) (2025)
- [MaskSQL: Safeguarding Privacy for LLM-Based Text-to-SQL via Abstraction](https://mlsanity.com/paper/2509.23459/) (2025)
- [Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practice](https://mlsanity.com/paper/2510.12595/) (2025)
- [ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Control](https://mlsanity.com/paper/2510.18232/) (2025)
- [Synthesizing and Adapting Error Correction Data for Mobile Large Language Model Applications](https://mlsanity.com/paper/2505.18488/) (2025)
- [Private Evolution Converges](https://mlsanity.com/paper/2506.08312/) (2025)
- [Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generation](https://mlsanity.com/paper/2604.07486/) (2026)
- [Differentially Private Synthetic Data via APIs 4: Tabular Data](https://mlsanity.com/paper/2606.08259/) (2026)
- [LanFL: Differentially Private Federated Learning with Large Language Models using Synthetic Samples](https://mlsanity.com/paper/2410.19114/) (2024)
- [FedRW: Efficient Privacy-Preserving Data Reweighting for Enhancing Federated Learning of Language Models](https://mlsanity.com/paper/2511.07505/) (2025)
- [Differentially Private Preference Data Synthesis for Large Language Model Alignment](https://mlsanity.com/paper/2605.30808/) (2026)
- [FedTDP: A Privacy-Preserving and Unified Framework for Trajectory Data Preparation via Federated Learning](https://mlsanity.com/paper/2505.05155/) (2025)
- [LLM-QFL: Distilling Large Language Model for Quantum Federated Learning](https://mlsanity.com/paper/2505.18656/) (2025)
- [Private Training & Data Generation by Clustering Embeddings](https://mlsanity.com/paper/2506.16661/) (2025)
- [Resource-Adaptive Federated Text Generation with Differential Privacy](https://mlsanity.com/paper/2603.07027/) (2026)
- [MAPLE: Metadata Augmented Private Language Evolution](https://mlsanity.com/paper/2603.19258/) (2026)
- [Jacobian-Guided Anisotropic Noise Reshaping for Enhancing Representation Utility under Local Differential Privacy](https://mlsanity.com/paper/2605.16812/) (2026)
- [ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?](https://mlsanity.com/paper/2606.01849/) (2026)
- [Collaborative Synthetic Data Generation for Knowledge Transfer in Federated Learning](https://mlsanity.com/paper/2607.07565/) (2026)
- [Canaries in the Bank: Auditing User-Level Privacy in Private Evolution](https://mlsanity.com/paper/2609.13499/) (2026)
- [QuanText: Protecting Dataset-Level Secrets in Textual Data Sharing](https://mlsanity.com/paper/2609.17995/) (2026)
- [Only Pay What You Must Spend: On-Demand Privacy Budget Payment for Differentially Private RAG](https://mlsanity.com/paper/2609.27406/) (2026)

## Abstract (arXiv)

On-device training is currently the most common approach for training machine learning (ML) models on private, distributed user data. Despite this, on-device training has several drawbacks: (1) most user devices are too small to train large models on-device, (2) on-device training is communication- and computation-intensive, and (3) on-device training can be difficult to debug and deploy. To address these problems, we propose Private Evolution-Text (PrE-Text), a method for generating differentially private (DP) synthetic textual data. First, we show that across multiple datasets, training small models (models that fit on user devices) with PrE-Text synthetic data outperforms small models trained on-device under practical privacy regimes (ε=1.29, ε=7.58). We achieve these results while using 9× fewer rounds, 6× less client computation per round, and 100× less communication per round. Second, finetuning large models on PrE-Text's DP synthetic data improves large language model (LLM) performance on private data across the same range of privacy budgets. Altogether, these results suggest that training on DP synthetic data can be a better option than training a model on-device on private distributed data. Code is available at https://github.com/houcharlie/PrE-Text.