mlsanityData to 14 Oct 2026

FL-NTK: A Neural Tangent Kernel-based Framework for Federated Learning Convergence Analysis

arXiv 2105.05001 ↗First posted 11 May 2021cs.LG cs.DC stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) is an emerging learning scheme that allows different distributed clients to train deep neural networks together without data sharing. Neural networks have become popular due to their unprecedented success. To the best of our knowledge, the theoretical guarantees of FL concerning neural networks with explicit forms and multi-step updates are unexplored. Nevertheless, training analysis of neural networks in FL is non-trivial for two reasons: first, the objective loss function we are optimizing is non-smooth and non-convex, and second, we are even not updating in the gradient direction. Existing convergence results for gradient descent-based methods heavily rely on the fact that the gradient direction is used for updating. This paper presents a new class of convergence analysis for FL, Federated Learning Neural Tangent Kernel (FL-NTK), which corresponds to overparamterized ReLU neural networks trained by gradient descent in FL and is inspired by the analysis in Neural Tangent Kernel (NTK). Theoretically, FL-NTK converges to a global-optimal solution at a linear rate with properly tuned learning parameters. Furthermore, with proper distributional assumptions, FL-NTK can also achieve good generalization.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 11 May 2021.

Builds on

35 of 48 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  7. 2018Federated Learning with Non-IID Datacited by 1,108
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  9. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  10. 2020The Future of Digital Health with Federated Learningcited by 633
  11. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
23 more
  1. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  2. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  3. 2020Federated Learning with Matched Averagingcited by 564
  4. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 409
  5. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 408
  6. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  7. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 300
  8. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  9. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  10. 2019A Joint Learning and Communications Framework for Federated Learning over Wireless Networkscited by 263
  11. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  12. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  13. 2019Privacy-preserving Federated Brain Tumour Segmentationcited by 169
  14. 2020Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Resultscited by 122
  15. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  16. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  17. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  18. 2020Siloed Federated Learning for Multi-Centric Histopathology Datasetscited by 61
  19. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 59
  20. 2020Training (Overparametrized) Neural Networks in Near-Linear Timecited by 58
  21. 2019Federated Transfer Reinforcement Learning for Autonomous Drivingcited by 44
  22. 2020Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionalitycited by 36
  23. 2020Generalized Leverage Score Sampling for Neural Networkscited by 26

Built on by

Cited by year

Citing papers per year: 2021: 8; 2022: 2; 2023: 3.

13 papers.

Sort:

  1. 2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
  2. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 56
  3. 2021Fast Sketching of Polynomial Kernels of Polynomial Degreecited by 33
  4. 2021Does Preprocessing Help Training Over-parameterized Neural Networks?cited by 32
  5. 2022Bypass Exponential Time Preprocessing: Fast Neural Network Training via Weight-Data Correlation Preprocessingcited by 29
  6. 2021Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learningcited by 23
  7. 2023Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networkscited by 16
  8. 2021Neural Tangent Kernel Empowered Federated Learningcited by 15
  9. 2021On Convergence of Federated Averaging Langevin Dynamicscited by 14
  10. 2021On the Convergence of Shallow Neural Network Training with Randomly Masked Neuronscited by 11
  11. 2023An effective theory of collective deep learningcited by 7
  12. 2021Federated Learning via Plurality Votecited by 6
1 more
  1. 2023Convergence of Two-Layer Regression with Nonlinear Unitscited by 6