mlsanityData to 13 Oct 2026

A Statistical Analysis of Deep Federated Learning for Intrinsically Low-dimensional Data

arXiv 2410.20659 ↗First posted 28 Oct 2024stat.ML cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Despite significant research on the optimization aspects of federated learning, the exploration of generalization error, especially in the realm of heterogeneous federated learning, remains an area that has been insufficiently investigated, primarily limited to developments in the parametric regime. This paper delves into the generalization properties of deep federated regression within a two-stage sampling model. Our findings reveal that the intrinsic dimension, characterized by the entropic dimension, plays a pivotal role in determining the convergence rates for deep learners when appropriately chosen network sizes are employed. Specifically, when the true relationship between the response and explanatory variables is described by a β-H"older function and one has access to n independent and identically distributed (i.i.d.) samples from m participating clients, for participating clients, the error rate scales at most as TildeO((mn)^-2β/(2β + bard2β(λ))), whereas for non-participating clients, it scales as TildeO(Delta · m^-2β/(2β + bard2β(λ)) + (mn)^-2β/(2β + bard2β(λ))). Here bard2β(λ) denotes the corresponding 2β-entropic dimension of λ, the marginal distribution of the explanatory variables. The dependence between the two stages of the sampling scheme is characterized by Delta. Consequently, our findings not only explicitly incorporate the ``heterogeneity" of the clients, but also highlight that the convergence rates of errors of deep federated learners are not contingent on the nominal high dimensionality of the data but rather on its intrinsic dimension.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 42 of its 63 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

42 of 63 references.

References · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,653
  3. 2019Advances and Open Problems in Federated Learningcited by 3,040
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,981
  5. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,619
  6. 2021Model-Contrastive Federated Learningcited by 766
  7. 2016Error bounds for approximations with deep ReLU networkscited by 570
  8. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 502
  9. 2019Agnostic Federated Learningcited by 453
  10. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 432
  11. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  12. 2016Adversarial Feature Learningcited by 350
30 more
  1. 2021A Field Guide to Federated Optimizationcited by 314
  2. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 312
  3. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 221
  4. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 178
  5. 2020Deep Network Approximation for Smooth Functionscited by 163
  6. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 161
  7. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  8. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  9. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  10. 2021FairFed: Enabling Group Fairness in Federated Learningcited by 117
  11. 2020Client Selection and Bandwidth Allocation in Wireless Federated Learning Networks: A Long-Term Perspectivecited by 102
  12. 2020Robust Federated Learning: The Case of Affine Distribution Shiftscited by 102
  13. 2023Federated Learning for Medical Image Analysis: A Surveycited by 78
  14. 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
  15. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  16. 2019Age-Based Scheduling Policy for Federated Learning in Mobile Edge Networkscited by 39
  17. 2019Nonlinear Approximation via Compositionscited by 37
  18. 2021An error analysis of generative adversarial networks for learning distributionscited by 36
  19. 2021Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyondcited by 36
  20. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  21. 2021A New Look and Convergence Rate of Federated Multi-Task Learning with Laplacian Regularizationcited by 27
  22. 2023Understanding Generalization of Federated Learning via Stability: Heterogeneity Matterscited by 27
  23. 2020Distribution Approximation and Statistical Estimation Guarantees of Generative Adversarial Networkscited by 23
  24. 2016Local Rademacher Complexity-based Learning Guarantees for Multi-Task Learningcited by 16
  25. 2021Faster Rates for Compressed Federated Learning with Client-Variance Reductioncited by 14
  26. 2023On Deep Generative Models for Approximation and Estimation of Distributions on Manifoldscited by 13
  27. 2024On the Statistical Properties of Generative Adversarial Models for Low Intrinsic Data Dimensioncited by 11
  28. 2024Communication-Efficient Hybrid Federated Learning for E-health with Horizontal and Vertical Data Partitioningcited by 6
  29. 2024A Statistical Analysis of Wasserstein Autoencoders for Intrinsically Low-dimensional Datacited by 4
  30. 2023Federated Primal Dual Fixed Point Algorithmcited by 1

Built on by

Not cited yet.