mlsanityData to 14 Oct 2026

Multi-Level Branched Regularization for Federated Learning

arXiv 2207.06936 ↗First posted 14 Jul 2022cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

A critical challenge of federated learning is data heterogeneity and imbalance across clients, which leads to inconsistency between local networks and unstable convergence of global models. To alleviate the limitations, we propose a novel architectural regularization technique that constructs multiple auxiliary branches in each local model by grafting local and global subnetworks at several different levels and that learns the representations of the main pathway in the local model congruent to the auxiliary hybrid pathways via online knowledge distillation. The proposed technique is effective to robustify the global model even in the non-iid setting and is applicable to various federated learning frameworks conveniently without incurring extra communication costs. We perform comprehensive empirical studies and demonstrate remarkable performance gains in terms of accuracy and efficiency compared to existing methods. The source code is available at our project page.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 14 Jul 2022.

Builds on

31 of 39 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  9. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  10. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  11. 2018Federated Learning with Non-IID Datacited by 1,108
  12. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
19 more
  1. 2020Adaptive Federated Optimizationcited by 1,046
  2. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  3. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  4. 2021Model-Contrastive Federated Learningcited by 766
  5. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  6. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  7. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 409
  8. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  9. 2021A Field Guide to Federated Optimizationcited by 315
  10. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  11. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 175
  12. 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
  13. 2019First Analysis of Local GD on Heterogeneous Datacited by 81
  14. 2020Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithmscited by 78
  15. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  16. 2020FedDANE: A Federated Newton-Type Methodcited by 76
  17. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 60
  18. 2020Federated Knowledge Distillationcited by 51
  19. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 15; 2025: 8; 2026: 2.

34 papers.

Sort:

By area:

  1. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
  2. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
  3. 2023Eliminating Domain Bias for Federated Learning in Representation Spacecited by 38
  4. 2024Relaxed Contrastive Learning for Federated Learningcited by 32
  5. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  6. 2023GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgettingcited by 27
  7. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 24
  8. 2023Bold but Cautious: Unlocking the Potential of Personalized Federated Learning through Cautiously Aggressive Collaborationcited by 23
  9. 2023FedFA: Federated Feature Augmentationcited by 22
  10. 2024Federated Distillation: A Surveycited by 15
  11. 2024Personalized Federated Learning via Feature Distribution Adaptationcited by 13
  12. 2024Advances in Robust Federated Learning: A Survey with Heterogeneity Considerationscited by 12
22 more
  1. 2025Geometric Knowledge-Guided Localized Global Distribution Alignment for Federated Learningcited by 12
  2. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 9
  3. 2025Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generationcited by 8
  4. 2024Lightweight Unsupervised Federated Learning with Pretrained Vision Language Modelcited by 5
  5. 2024Exploring the Practicality of Federated Learning: A Survey Towards the Communication Perspectivecited by 4
  6. 2023Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectivescited by 3
  7. 2024The Diversity Bonus: Learning from Dissimilar Distributed Clients in Personalized Federated Learningcited by 3
  8. 2024FedQUIT: On-Device Federated Unlearning via a Quasi-Competent Virtual Teachercited by 2
  9. 2024Privacy-Preserving Federated Learning with Consistency via Knowledge Distillation Using Conditional Generatorcited by 2
  10. 2024DFDG: Data-Free Dual-Generator Adversarial Distillation for One-Shot Federated Learningcited by 2
  11. 2025Addressing Label Shift in Distributed Learning via Entropy Regularizationcited by 1
  12. 2025Federated Unlearning Made Practical: Seamless Integration via Negated Pseudo-Gradientscited by 1
  13. 2025FedWSQ: Efficient Federated Learning with Weight Standardization and Distribution-Aware Non-Uniform Quantizationcited by 1
  14. 2026Distribution Alignment for One-Shot Federated Learning via Optimal Transportcited by 1
  15. 2024DA-PFL: Dynamic Affinity Aggregation for Personalized Federated Learningcited by 0
  16. 2024Personalized Additive Modeling for Multi-level Federated Learningcited by 0
  17. 2024Fast convergence of a Federated Expectation-Maximization Algorithmcited by 0
  18. 2024Heterogeneity: An Open Challenge for Federated On-board Machine Learningcited by 0
  19. 2025ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoningcited by 0
  20. 2025DART: A Server-side Plug-in for Resource-efficient Robust Federated Learningcited by 0
  21. 2025Adversarial Versus Federated: An Adversarial Learning based Multi-Modality Cross-Domain Federated Medical Segmentationcited by 0
  22. 2026SDFed: Bridging Local Global Discrepancy via Subspace Refinement and Divergence Control in Federated Prompt Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Adaptive Self-Distillation for Minimizing Client Drift in Heterogeneous Federated Learning
  2. 2024Tailored Federated Learning: Leveraging Direction Regulation & Knowledge Distillation
  3. 2025Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data
  4. 2022Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion
  5. 2023Heterogeneous Federated Learning Using Knowledge Codistillation
  6. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning
  7. 2022Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
  8. 2023Federated Learning on Non-iid Data via Local and Global Distillation