mlsanityData to 14 Oct 2026

Share Your Representation Only: Guaranteed Improvement of the Privacy-Utility Tradeoff in Federated Learning

arXiv 2309.05505 ↗First posted 11 Sep 2023cs.LG stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Repeated parameter sharing in federated learning causes significant information leakage about private data, thus defeating its main purpose: data privacy. Mitigating the risk of this information leakage, using state of the art differentially private algorithms, also does not come for free. Randomized mechanisms can prevent convergence of models on learning even the useful representation functions, especially if there is more disagreement between local models on the classification functions (due to data heterogeneity). In this paper, we consider a representation federated learning objective that encourages various parties to collaboratively refine the consensus part of the model, with differential privacy guarantees, while separately allowing sufficient freedom for local personalization (without releasing it). We prove that in the linear representation setting, while the objective is non-convex, our proposed new algorithm DPFEDREP converges to a ball centered around the global optimal solution at a linear rate, and the radius of the ball is proportional to the reciprocal of the privacy budget. With this novel utility analysis, we improve the SOTA utility-privacy trade-off for this problem by a factor of √d, where d is the input dimension. We empirically evaluate our method with the image classification task on CIFAR10, CIFAR100, and EMNIST, and observe a significant performance improvement over the prior work under the same small privacy budget. The code can be found in this link: https://github.com/shenzebang/CENTAUR-Privacy-Federated-Representation-Learning.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

39 of 58 references.

References · 39

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2016Deep Learning with Differential Privacycited by 2,728
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  6. 2019Deep Leakage from Gradientscited by 1,021
  7. 2017Federated Multi-Task Learningcited by 672
  8. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  9. 2019Benign Overfitting in Linear Regressioncited by 549
  10. 2019Federated Learning with Personalization Layerscited by 541
  11. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  12. 2014RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Responsecited by 272
27 more
  1. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
  2. 2021Deep learning: a statistical viewpointcited by 240
  3. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  4. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
  5. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 204
  6. 2015The Benefit of Multitask Representation Learningcited by 195
  7. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
  8. 2020Provable Meta-Learning of Linear Representationscited by 161
  9. 2018cpSGD: Communication-efficient and differentially-private distributed SGDcited by 158
  10. 2017Collecting Telemetry Data Privatelycited by 149
  11. 2019Federated Evaluation of On-device Personalizationcited by 147
  12. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  13. 2022Federated Learning with Partial Model Personalizationcited by 114
  14. 2021Federated Reconstruction: Partially Local Federated Learningcited by 79
  15. 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
  16. 2021Learning with User-Level Privacycited by 72
  17. 2013The Noisy Power Method: A Meta Algorithm with Applicationscited by 58
  18. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 56
  19. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  20. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 30
  21. 2013Private Matchings and Allocationscited by 23
  22. 2022Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learningcited by 23
  23. 2022Learning to Generate Image Embeddings with User-level Differential Privacycited by 23
  24. 2022Towards Sample-efficient Overparameterized Meta-learningcited by 21
  25. 2014Improved Distributed Principal Component Analysiscited by 14
  26. 2021Private Multi-Task Learning: Formulation and Applications to Federated Learningcited by 11
  27. 2021Federated Functional Gradient Boostingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 5; 2026: 2.

13 papers.

Sort:

  1. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  2. 2024Clients Collaborate: Flexible Differentially Private Federated Learning with Guaranteed Improvement of Utility-Privacy Trade-offcited by 7
  3. 2024Federated Representation Learning in the Under-Parameterized Regimecited by 7
  4. 2023Private Learning with Public Featurescited by 6
  5. 2024Provable Mutual Benefits from Federated Learning in Privacy-Sensitive Domainscited by 2
  6. 2024Exploring Lightweight Federated Learning for Distributed Load Forecastingcited by 2
  7. 2025Differential Privacy Personalized Federated Learning Based on Dynamically Sparsified Client Updatescited by 0
  8. 2025A Linearized Alternating Direction Multiplier Method for Federated Matrix Completion Problemscited by 0
  9. 2025Black-Box Privacy Attacks on Shared Representations in Multitask Learningcited by 0
  10. 2025Learning Private Representations through Entropy-based Adversarial Trainingcited by 0
  11. 2025FedPF: Accurate Target Privacy Preserving Federated Learning Balancing Fairness and Utilitycited by 0
  12. 2026FedHypeVAE: Federated Learning with Hypernetwork Generated Conditional VAEs for Differentially Private Embedding Sharingcited by 0
1 more
  1. 2026A Taxonomy and Resolution Strategy for Client-Level Disagreements in Federated Learningcited by 0