Data to 14 Oct 2026

Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learning

arXiv 2308.11464 ↗First posted 22 Aug 2023cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) inevitably confronts the challenge of system heterogeneity in practical scenarios. To enhance the capabilities of most model-homogeneous FL methods in handling system heterogeneity, we propose a training scheme that can extend their capabilities to cope with this challenge. In this paper, we commence our study with a detailed exploration of homogeneous and heterogeneous FL settings and discover three key observations: (1) a positive correlation between client performance and layer similarities, (2) higher similarities in the shallow layers in contrast to the deep layers, and (3) the smoother gradients distributions indicate the higher layer similarities. Building upon these observations, we propose InCo Aggregation that leverages internal cross-layer gradients, a mixture of gradients from shallow and deep layers within a server model, to augment the similarity in the deep layers without requiring additional communication between clients. Furthermore, our methods can be tailored to accommodate model-homogeneous FL methods such as FedAvg, FedProx, FedNova, Scaffold, and MOON, to expand their capabilities to handle the system heterogeneity. Copious experimental results validate the effectiveness of InCo Aggregation, spotlighting internal cross-layer gradients as a promising avenue to enhance the performance in heterogeneous FL.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 47 of its 66 references that are in the map. First posted on arXiv 22 Aug 2023.

Builds on

47 of 66 references.

References · 47

2014Adam: A Method for Stochastic Optimizationcited by 37,240
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
2015Distilling the Knowledge in a Neural Networkcited by 10,324
2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
2019Advances and Open Problems in Federated Learningcited by 3,046
2018Federated Optimization in Heterogeneous Networkscited by 2,987
2021Convex optimizationcited by 2,515
2016Identity Mappings in Deep Residual Networkscited by 1,965
2019Similarity of Neural Network Representations Revisitedcited by 1,755
2020Knowledge Distillation: A Surveycited by 1,603
2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538

35 more

2018Federated Learning with Non-IID Datacited by 1,108
2016An overview of gradient descent optimization algorithmscited by 1,033
2020On Layer Normalization in the Transformer Architecturecited by 922
2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
2021Model-Contrastive Federated Learningcited by 766
2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
2019Agnostic Federated Learningcited by 456
2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
2018Insights on representational similarity in neural networks with canonical correlationcited by 324
2019Asynchronous Federated Optimizationcited by 267
2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
2019Universally Slimmable Networks and Improved Training Techniquescited by 221
2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
2019Data-Free Learning of Student Networkscited by 204
2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 163
2021Pretrained Transformers as Universal Computation Enginescited by 162
2019Zero-Shot Knowledge Distillation in Deep Networkscited by 121
2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
2020Federated Mutual Learningcited by 71
2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 69
2018Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representationcited by 62
2022A Survey on Heterogeneous Federated Learningcited by 42
2022No One Left Behind: Inclusive Federated Learning over Heterogeneous Devicescited by 37
2021FedH2L: Federated Learning with Model and Statistical Heterogeneitycited by 17
2021Joint Superposition Coding and Training for Federated Learning over Multi-Width Neural Networkscited by 14
2021FedHe: Heterogeneous Models and Communication-Efficient Federated Learningcited by 13
2021Gaussian RBF Centered Kernel Alignment (CKA) in the Large Bandwidth Limitcited by 6
2022Exploiting Features and Logits in Heterogeneous Federated Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
2024Federated Model Heterogeneous Matryoshka Representation Learningcited by 21
2025Federated Learning with Sample-level Client Drift Mitigationcited by 6
2025TACO: Tackling Over-correction in Federated Learning with Tailored Adaptive Correctioncited by 0
2026SpecGradFilter: A Spectral Gradient Filtering Framework for Taming Federated Heterogeneitycited by 0