mlsanityData to 14 Oct 2026

POPri: Private Federated Learning using Preference-Optimized Synthetic Data

arXiv 2504.16438 ↗First posted 23 Apr 2025cs.LG cs.AI cs.CRcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

also in 1 more
Abstract (arXiv)

In practical settings, differentially private Federated learning (DP-FL) is the dominant method for training models from private, on-device client data. Recent work has suggested that DP-FL may be enhanced or outperformed by methods that use DP synthetic data (Wu et al., 2024; Hou et al., 2024). The primary algorithms for generating DP synthetic data for FL applications require careful prompt engineering based on public information and/or iterative private client feedback. Our key insight is that the private client feedback collected by prior DP synthetic data methods (Hou et al., 2024; Xie et al., 2024) can be viewed as an RL (reinforcement learning) reward. Our algorithm, Policy Optimization for Private Data (POPri) harnesses client feedback using policy optimization algorithms such as Direct Preference Optimization (DPO) to fine-tune LLMs to generate high-quality DP synthetic data. To evaluate POPri, we release LargeFedBench, a new federated text benchmark for uncontaminated LLM evaluations on federated client data. POPri substantially improves the utility of DP synthetic data relative to prior work on LargeFedBench datasets and an existing benchmark from Xie et al. (2024). POPri closes the gap between next-token prediction accuracy in the fully-private and non-private settings by up to 58%, compared to 28% for prior synthetic data methods, and 3% for state-of-the-art DP federated learning methods. The code and data are available at https://github.com/meiyuw/POPri.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 62 references that are in the map. First posted on arXiv 23 Apr 2025.

Builds on

42 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Privacy-Preserving Instructions for Aligning Large Language Models21 Feb

Other references · 41

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2019Advances and Open Problems in Federated Learningcited by 3,046
  10. 2016Deep Learning with Differential Privacycited by 2,728
  11. 2023PaLM 2 Technical Reportcited by 1,162
  12. 2020Adaptive Federated Optimizationcited by 1,046
29 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  2. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  3. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
  4. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  5. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  6. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 195
  7. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  8. 2016Practical Secure Aggregation for Federated Learning on User-Held Datacited by 178
  9. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
  10. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  11. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  12. 2021Federated Evaluation and Tuning for On-Device Personalization: System Design & Applicationscited by 85
  13. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 76
  14. 2022Differentially Private Language Models for Secure Data Sharingcited by 68
  15. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 66
  16. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 65
  17. 2023Harnessing large-language models to generate private synthetic textcited by 65
  18. 2023Locally Differentially Private Document Generation Using Zero Shot Promptingcited by 63
  19. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 30
  20. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  21. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  22. 2024Fine-Tuning Large Language Models with User-Level Differential Privacycited by 29
  23. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  24. 2022Learning to Generate Image Embeddings with User-level Differential Privacycited by 23
  25. 2022On the Privacy Properties of GAN-generated Samplescited by 17
  26. 2024Prompt Public Large Language Models to Synthesize Data for Private On-device Applicationscited by 17
  27. 2025Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Modelcited by 13
  28. 2025Contrastive Private Data Synthesis via Weighted Multi-PLM Fusioncited by 11
  29. 2023Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2025Struct-Bench: A Benchmark for Differentially Private Structured Text Generationcited by 7
  2. 2025ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Controlcited by 3
  3. 2025Private Evolution Convergescited by 2
  4. 2026EPSVec: Efficient and Private Synthetic Data Generation via Dataset Vectorscited by 2
  5. 2026Differentially Private Synthetic Data via APIs 4: Tabular Datacited by 2
  6. 2026A Step Toward Federated Pretraining of Multimodal Large Language Modelscited by 1
  7. 2025Secret-Protected Evolution for Differentially Private Synthetic Text Generationcited by 0
  8. 2026DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuningcited by 0
  9. 2026Resource-Adaptive Federated Text Generation with Differential Privacycited by 0
  10. 2026MAPLE: Metadata Augmented Private Language Evolutioncited by 0
  11. 2026PubSwap: Public-Data Off-Policy Coordination for Federated RLVRcited by 0
  12. 2026Concordia: Self-Improving Synthetic Tables for Federated LLMscited by 0
2 more
  1. 2026FedEHR-Gen: Federated Synthetic Time-Series EHR Generation via Latent Space Alignment and Distribution-Aware Aggregationcited by 0
  2. 2026Private Direct Preference Optimization for LLM Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Synthetic Data Aided Federated Learning Using Foundation Models
  2. 2025Federated Learning: A Survey on Privacy-Preserving Collaborative Intelligence
  3. 2024Privacy in Federated Learning
  4. 2022Federated Learning with GAN-based Data Synthesis for Non-IID Clients
  5. 2026FedGRPO: Privately Optimizing Foundation Models with Group-Relative Rewards from Domain Client
  6. 2025Personalized Federated Training of Diffusion Models with Privacy Guarantees
  7. 2021Personalised Federated Learning: A Combinational Approach
  8. 2025FedRand: Enhancing Privacy in Federated Learning with Randomized LoRA Subparameter Updates