mlsanityData to 14 Oct 2026

Neural Tangent Kernel Empowered Federated Learning

arXiv 2110.03681 ↗First posted 7 Oct 2021cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is a privacy-preserving paradigm where multiple participants jointly solve a machine learning problem without sharing raw data. Unlike traditional distributed learning, a unique characteristic of FL is statistical heterogeneity, namely, data distributions across participants are different from each other. Meanwhile, recent advances in the interpretation of neural networks have seen a wide use of neural tangent kernels (NTKs) for convergence analyses. In this paper, we propose a novel FL paradigm empowered by the NTK framework. The paradigm addresses the challenge of statistical heterogeneity by transmitting update data that are more expressive than those of the conventional FL paradigms. Specifically, sample-wise Jacobian matrices, rather than model weights/gradients, are uploaded by participants. The server then constructs an empirical kernel matrix to update a global model without explicitly performing gradient descent. We further develop a variant with improved communication efficiency and enhanced privacy. Numerical results show that the proposed paradigm can achieve the same accuracy while reducing the number of communication rounds by an order of magnitude compared to federated averaging.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 7 Oct 2021.

Builds on

38 of 51 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  7. 2018Federated Learning with Non-IID Datacited by 1,108
  8. 2020Adaptive Federated Optimizationcited by 1,046
  9. 2019Deep Leakage from Gradientscited by 1,021
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  11. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  12. 2018LEAF: A Benchmark for Federated Settingscited by 792
26 more
  1. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  2. 2017Federated Multi-Task Learningcited by 672
  3. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  5. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  6. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  7. 2020Federated Learning with Matched Averagingcited by 564
  8. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  9. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  10. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
  11. 2021A Field Guide to Federated Optimizationcited by 315
  12. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  13. 2019The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networkscited by 246
  14. 2018The Convergence of Sparsified Gradient Methodscited by 229
  15. 2018Machine Learning with Membership Privacy using Adversarial Regularizationcited by 186
  16. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  17. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 160
  18. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  19. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  20. 2020The Recurrent Neural Tangent Kernelcited by 67
  21. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  22. 2020Federated Knowledge Distillationcited by 51
  23. 2020Optimization Theory for ReLU Neural Networks Trained with Normalization Layerscited by 15
  24. 2021FL-NTK: A Neural Tangent Kernel-based Framework for Federated Learning Convergence Analysiscited by 13
  25. 2021Separation of Powers in Federated Learningcited by 8
  26. 2020Federated Learning of User Authentication Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 3; 2024: 6; 2025: 2; 2026: 1.

15 papers.

Sort:

  1. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
  2. 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 69
  3. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  4. 2022Neural Tangent Kernel: A Surveycited by 23
  5. 2024FedFisher: Leveraging Fisher Information for One-Shot Federated Learningcited by 15
  6. 2022Anchor Sampling for Federated Learning with Partial Client Participationcited by 14
  7. 2024A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applicationscited by 8
  8. 2024Federated Learning Resilient to Byzantine Attacks and Data Heterogeneitycited by 4
  9. 2024NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernelcited by 4
  10. 2024Efficient Federated Learning against Byzantine Attacks and Data Heterogeneity via Aggregating Normalized Gradientscited by 2
  11. 2025Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvgcited by 2
  12. 2024Sequential Federated Learning in Hierarchical Architecture on Non-IID Datasetscited by 1
3 more
  1. 2023Factor-Assisted Federated Learning for Personalized Optimization with Heterogeneous Datacited by 0
  2. 2025Communication-Efficient Neural Tangent Kernels for Heterogeneous Decentralized Federated Learningcited by 0
  3. 2026Beyond Parameter Space: NTK-Guided Personalized Aggregation for Robust Federated Learningcited by 0