# TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernels

arXiv: https://arxiv.org/abs/2207.06343 · HTML: https://mlsanity.com/paper/2207.06343/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 89 references that are in the map.
- First posted on arXiv 13 Jul 2022.

## Builds on (most cited first, 30 of 65)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [How To Backdoor Federated Learning](https://mlsanity.com/paper/1807.00459/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [Local Model Poisoning Attacks to Byzantine-Robust Federated Learning](https://mlsanity.com/paper/1911.11815/) (2019)
- [Agnostic Federated Learning](https://mlsanity.com/paper/1902.00146/) (2019)
- [Adaptive Federated Learning in Resource Constrained Edge Computing Systems](https://mlsanity.com/paper/1804.05271/) (2018)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [Fair Resource Allocation in Federated Learning](https://mlsanity.com/paper/1905.10497/) (2019)
- [Adaptive Personalized Federated Learning](https://mlsanity.com/paper/2003.13461/) (2020)
- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Personalized Federated Learning: A Meta-Learning Approach](https://mlsanity.com/paper/2002.07948/) (2020)

## Built on by (most cited first, 26 of 26)

- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [FedExP: Speeding Up Federated Averaging via Extrapolation](https://mlsanity.com/paper/2301.09604/) (2023)
- [Dataset Distillation with Convexified Implicit Gradients](https://mlsanity.com/paper/2302.06755/) (2023)
- [Federated Conformal Predictors for Distributed Uncertainty Quantification](https://mlsanity.com/paper/2305.17564/) (2023)
- [Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning](https://mlsanity.com/paper/2310.18285/) (2023)
- [A Survey for Federated Learning Evaluations: Goals and Measures](https://mlsanity.com/paper/2308.11841/) (2023)
- [FedFisher: Leveraging Fisher Information for One-Shot Federated Learning](https://mlsanity.com/paper/2403.12329/) (2024)
- [Accelerating Heterogeneous Federated Learning with Closed-form Classifiers](https://mlsanity.com/paper/2406.01116/) (2024)
- [On the effectiveness of partial variance reduction in federated learning with heterogeneous data](https://mlsanity.com/paper/2212.02191/) (2022)
- [Privacy Preserving Federated Learning in Medical Imaging with Uncertainty Estimation](https://mlsanity.com/paper/2406.12815/) (2024)
- [Scaff-PD: Communication Efficient Fair and Robust Federated Learning](https://mlsanity.com/paper/2307.13381/) (2023)
- [DAVED: Data Acquisition via Experimental Design for Data Markets](https://mlsanity.com/paper/2403.13893/) (2024)
- [FedSAC: Dynamic Submodel Allocation for Collaborative Fairness in Federated Learning](https://mlsanity.com/paper/2405.18291/) (2024)
- [Exploring the Practicality of Federated Learning: A Survey Towards the Communication Perspective](https://mlsanity.com/paper/2405.20431/) (2024)
- [NTK-DFL: Enhancing Decentralized Federated Learning in Heterogeneous Settings via Neural Tangent Kernel](https://mlsanity.com/paper/2410.01922/) (2024)
- [Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity](https://mlsanity.com/paper/2306.13263/) (2023)
- [The Power of Bias: Optimizing Client Selection in Federated Learning with Heterogeneous Differential Privacy](https://mlsanity.com/paper/2408.08642/) (2024)
- [Future-Proofing Medical Imaging with Privacy-Preserving Federated Learning and Uncertainty Quantification: A Review](https://mlsanity.com/paper/2409.16340/) (2024)
- [Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvg](https://mlsanity.com/paper/2508.12576/) (2025)
- [FedSat: A Statistical Aggregation Approach for Class Imbalanced Clients in Federated Learning](https://mlsanity.com/paper/2407.03862/) (2024)
- [Federated Learning with Relative Fairness](https://mlsanity.com/paper/2411.01161/) (2024)
- [Convergence Analysis of Federated Learning Methods Using Backward Error Analysis](https://mlsanity.com/paper/2503.03139/) (2025)
- [Communication-Efficient Neural Tangent Kernels for Heterogeneous Decentralized Federated Learning](https://mlsanity.com/paper/2512.12737/) (2025)
- [Federated Ensemble Learning with Progressive Model Personalization](https://mlsanity.com/paper/2602.14244/) (2026)
- [FLAM: Evaluating Model Performance with Aggregatable Measures in Federated Learning](https://mlsanity.com/paper/2605.07962/) (2026)
- [How Edge of Stability Hinders SCAFFOLD in Federated Optimization](https://mlsanity.com/paper/2608.25873/) (2026)

## Abstract (arXiv)

State-of-the-art federated learning methods can perform far worse than their centralized counterparts when clients have dissimilar data distributions. For neural networks, even when centralized SGD easily finds a solution that is simultaneously performant for all clients, current federated optimization methods fail to converge to a comparable solution. We show that this performance disparity can largely be attributed to optimization challenges presented by nonconvexity. Specifically, we find that the early layers of the network do learn useful features, but the final layers fail to make use of them. That is, federated optimization applied to this non-convex problem distorts the learning of the final layers. Leveraging this observation, we propose a Train-Convexify-Train (TCT) procedure to sidestep this issue: first, learn features using off-the-shelf methods (e.g., FedAvg); then, optimize a convexified problem obtained from the network's empirical neural tangent kernel approximation. Our technique yields accuracy improvements of up to +36% on FMNIST and +37% on CIFAR10 when clients have dissimilar data.