# Resource-Adaptive Federated Text Generation with Differential Privacy

arXiv: https://arxiv.org/abs/2603.07027 · HTML: https://mlsanity.com/paper/2603.07027/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 7 Mar 2026.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Federated Learning with Differential Privacy: Algorithms and Performance Analysis](https://mlsanity.com/paper/1911.00222/) (2019)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoising](https://mlsanity.com/paper/1805.06530/) (2018)
- [Federated Learning with Buffered Asynchronous Aggregation](https://mlsanity.com/paper/2106.06639/) (2021)
- [Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentation](https://mlsanity.com/paper/1810.04304/) (2018)
- [FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations](https://mlsanity.com/paper/2409.05976/) (2024)
- [Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe](https://mlsanity.com/paper/2210.14348/) (2022)
- [Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources](https://mlsanity.com/paper/2402.11505/) (2024)
- [Cross-Silo Federated Learning: Challenges and Opportunities](https://mlsanity.com/paper/2206.12949/) (2022)
- [Can Public Large Language Models Help Private Cross-device Federated Learning?](https://mlsanity.com/paper/2305.12132/) (2023)
- [PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs](https://mlsanity.com/paper/2406.02958/) (2024)
- [A Survey on Federated Fine-tuning of Large Language Models](https://mlsanity.com/paper/2503.12016/) (2025)
- [Prompt Public Large Language Models to Synthesize Data for Private On-device Applications](https://mlsanity.com/paper/2404.04360/) (2024)
- [Exploring Distributional Shifts in Large Language Models for Code Analysis](https://mlsanity.com/paper/2303.09128/) (2023)
- [POPri: Private Federated Learning using Preference-Optimized Synthetic Data](https://mlsanity.com/paper/2504.16438/) (2025)
- [Ask Your Distribution Shift if Pre-Training is Right for You](https://mlsanity.com/paper/2403.00194/) (2024)
- [HLoRA: Efficient Federated Learning System for LLM Heterogeneous Fine-Tuning](https://mlsanity.com/paper/2503.00813/) (2025)

## Abstract (arXiv)

In cross-silo federated learning (FL), sensitive text datasets remain confined to local organizations due to privacy regulations, making repeated training for each downstream task both communication-intensive and privacy-demanding. A promising alternative is to generate differentially private (DP) synthetic datasets that approximate the global distribution and can be reused across tasks. However, pretrained large language models (LLMs) often fail under domain shift, and federated finetuning is hindered by computational heterogeneity: only resource-rich clients can update the model, while weaker clients are excluded, amplifying data skew and the adverse effects of DP noise. We propose a flexible participation framework that adapts to client capacities. Strong clients perform DP federated finetuning, while weak clients contribute through a lightweight DP voting mechanism that refines synthetic text. To ensure the synthetic data mirrors the global dataset, we apply control codes (e.g., labels, topics, metadata) that represent each client's data proportions and constrain voting to semantically coherent subsets. This two-phase approach requires only a single round of communication for weak clients and integrates contributions from all participants. Experiments show that our framework improves distribution alignment and downstream robustness under DP and heterogeneity.