mlsanityData to 7 Oct 2026

Privately Customizing Prefinetuning to Better Match User Data in Federated Learning

arXiv 2302.09042 ↗First posted 17 Feb 2023cs.LG cs.AI cs.DCcited by 12

Not on the map. Why?

Abstract (arXiv)

In Federated Learning (FL), accessing private client data incurs communication and privacy costs. As a result, FL deployments commonly prefinetune pretrained foundation models on a (large, possibly public) dataset that is held by the central server; they then FL-finetune the model on a private, federated dataset held by clients. Evaluating prefinetuning dataset quality reliably and privately is therefore of high importance. To this end, we propose FreD (Federated Private Fr'echet Distance) -- a privately computed distance between a prefinetuning dataset and federated datasets. Intuitively, it privately computes and compares a Fr'echet distance between embeddings generated by a large language model on both the central (public) dataset and the federated private client data. To make this computation privacy-preserving, we use distributed, differentially-private mean and covariance estimators. We show empirically that FreD accurately predicts the best prefinetuning dataset at minimal privacy cost. Altogether, using FreD we demonstrate a proof-of-concept for a new approach in private FL training: (1) customize a prefinetuning dataset to better match user data (2) prefinetune (3) perform FL-finetuning.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 15 of its 23 references that are in the map. First posted on arXiv 17 Feb 2023.

Builds on

15 of 23 references.

References · 15

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2018LEAF: A Benchmark for Federated Settingscited by 780
  10. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  11. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  12. 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
3 more
  1. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
  2. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  3. 2021Assessing Dialogue Systems with Distribution Distancescited by 16

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 6; 2025: 2.

12 papers.

  1. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  2. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  3. 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24
  4. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  5. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  6. 2023Selective Pre-training for Private Fine-tuningcited by 22
  7. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  8. 2024Prompt Public Large Language Models to Synthesize Data for Private On-device Applicationscited by 17
  9. 2024Efficiently Computing Similarities to Private Datasetscited by 7
  10. 2025AugFL: Augmenting Federated Learning with Pretrained Modelscited by 6
  11. 2024Differentially Private Kernel Density Estimationcited by 2
  12. 2024Federated and differentially private estimation of KL divergencecited by 0