Federated Learning Based on Dynamic Regularization
Not on the map. Why?
Abstract (arXiv)
We propose a novel federated learning method for distributively training neural network models, where the server orchestrates cooperation between a subset of randomly chosen devices in each round. We view Federated Learning problem primarily from a communication perspective and allow more device level computations to save transmission costs. We point out a fundamental dilemma, in that the minima of the local-device level empirical loss are inconsistent with those of the global empirical loss. Different from recent prior works, that either attempt inexact minimization or utilize devices for parallelizing gradient computation, we propose a dynamic regularizer for each device at each round, so that in the limit the global and device solutions are aligned. We demonstrate both through empirical results on real and synthetic data as well as analytical results that our scheme leads to efficient training, in both convex and non-convex settings, while being fully agnostic to device heterogeneity and robust to large number of devices, partial participation and unbalanced data.
Key facts. Cited by 555 papers in the map (data through 7 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 8 Nov 2021.
Builds on
32 of 41 references.
References · 32
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
- 2018Group Normalizationcited by 1,514
- 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
- 2018Federated Learning with Non-IID Datacited by 1,099
- 2020Adaptive Federated Optimizationcited by 1,036
- 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
- 2018LEAF: A Benchmark for Federated Settingscited by 780
- 2018Local SGD Converges Fast and Communicates Littlecited by 595
20 more
- 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
- 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
- 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
- 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
- 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
- 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 143
- 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 131
- 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
- 2020Acceleration for Compressed Gradient Descent in Distributed and Federated Optimizationcited by 90
- 2020Federated Accelerated Stochastic Gradient Descentcited by 85
- 2019A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descentcited by 84
- 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
- 2020FedDANE: A Federated Newton-Type Methodcited by 76
- 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
- 2020Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimizationcited by 36
- 2019On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learningcited by 31
- 2020A Unified Analysis of Stochastic Gradient Methods for Nonconvex Federated Optimizationcited by 27
- 2016Convergence Rate of Distributed ADMM over Networkscited by 16
- 2018Primal-dual accelerated gradient methods with small-dimensional relaxation oraclecited by 14
- 2020Distributed Proximal Splitting Algorithms with Rates and Accelerationcited by 7
Built on by
Citing papers per year: 2021: 7; 2022: 88; 2023: 143; 2024: 128; 2025: 108; 2026: 81.
555 papers.
Sort:
By area:
- 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
- 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
- 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
- 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
- 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
- 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
- 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 108
- 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
- 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
- 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
- 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
- 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
38 more
- 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
- 2021Differentially Private Federated Learning on Heterogeneous Datacited by 74
- 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
- 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 73
- 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
- 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 68
- 2022Tackling Data Heterogeneity in Federated Learning with Class Prototypescited by 66
- 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 66
- 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 62
- 2022FedCorr: Multi-Stage Federated Learning for Label Noise Correctioncited by 61
- 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
- 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 57
- 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
- 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
- 2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
- 2022FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Datacited by 49
- 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
- 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
- 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
- 2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 45
- 2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
- 2022FedDrive: Generalizing Federated Learning to Semantic Segmentation in Autonomous Drivingcited by 41
- 2022Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentationcited by 41
- 2023Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challengescited by 39
- 2023Eliminating Domain Bias for Federated Learning in Representation Spacecited by 38
- 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 37
- 2022Differentially Private Federated Learning with Local Regularization and Sparsificationcited by 35
- 2022FedADMM: A Federated Primal-Dual Algorithm Allowing Partial Participationcited by 35
- 2022Learning Across Domains and Devices: Style-Driven Source-Free Domain Adaptation in Clustered Federated Learningcited by 35
- 2022SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compressioncited by 34
- 2022Multi-Level Branched Regularization for Federated Learningcited by 34
- 2021Accurate online training of dynamical spiking neural networks through Forward Propagation Through Timecited by 33
- 2024An Aggregation-Free Federated Learning for Tackling Data Heterogeneitycited by 33
- 2024Relaxed Contrastive Learning for Federated Learningcited by 32
- 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
- 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
- 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
- 2022Architecture Agnostic Federated Learning for Neural Networkscited by 30
50 most cited of 555 All 555 (JSON)
- 2026FedCKA: Representation-Guided Layer Personalization for Federated 3D Perception Across Driving Domains1 Oct 2026
- 2026Latent Information Sharing for Accelerating Federated Learning1 Oct 2026
- 2026A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilization21 Sep 2026
- 2026Joint Domain-Class Modeling for Federated Learning Under Feature Skew19 Sep 2026
- 2026FedeRICo: Federated Region-Influenced Coupling for Traffic Flow Prediction17 Sep 2026
- 2026Structural Negative Transfer in Federated Graph Neural Networks: Diagnosis, Causal Investigation, and the Limits of Divergence-Aware Mitigation15 Sep 2026
- 2026Adaptive Bayesian Partner Selection for Federated Clinical Centers15 Sep 2026
- 2026CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimization14 Sep 2026
- 2026FedDRAW: Federated Dual Reputation Annealing Weighting for Heterogeneous Multi-Institutional Chest Radiograph Classification4 Sep 2026
- 2026FedCMAPSS: A Benchmark for Federated Learning in Remaining Useful Life Estimation26 Aug 2026
- 2026A Momentum-Based Variance-Reduced Algorithm for Federated Multiobjective Optimization24 Aug 2026
- 2026FedADB: Class Anchor-Driven Dual-Branch Federated Learning for Mitigating Forgetting15 Aug 2026
38 more
- 2026Cluster-Aware Over-the-Air Federated Learning with Energy-Harvesting Devices: From Global Training to Model Personalization2 Aug 2026
- 2026FeDepth: Federated Learning for Depth Estimation under Robot Heterogeneity2 Aug 2026
- 2026FL-OA: A Byzantine-Robust Federated Learning Framework with Outsourced Auditing for Intelligent Devices2 Aug 2026
- 2026FedImp: Enhancing Federated Learning Convergence with Impurity-Based Weighting31 Jul 2026
- 2026TriShield: Zero-Utility-Loss Defense Against Privacy Backdoors in Federated Language Model Fine-Tuning via Orthogonal Gradient Projection and Optimizer State Entanglement30 Jul 2026
- 2026Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers23 Jul 2026
- 2026One Round Is All You Need: Analytic Federated Learning for Task-Heterogeneous Multi-Label Medical Image Classification22 Jul 2026
- 2026FedASAP: Activation Statistics-driven Structured Adaptive Pruning for Efficient Personalized Federated Learning for Lesion Segmentation on brain MRI20 Jul 2026
- 2026A Framework for Early Sepsis Prediction via Self-Supervised (JEPA) and Federated Representation Learning18 Jul 2026
- 2026FM2: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging15 Jul 2026
- 2026Benchmark Evaluation of Federated Learning on Multi-organ Images9 Jul 2026
- 2026WHERE to Generate Matters: Budget-Aware Synthetic Augmentation for Label Skewed Federated Learning7 Jul 2026
- 2026SpecGradFilter: A Spectral Gradient Filtering Framework for Taming Federated Heterogeneity5 Jul 2026
- 2026FedFFT: Taming Client Drift in Federated SAM via Spectral Perturbation Filtering5 Jul 2026
- 2026FedProIn: Mitigating Client Drift for Learnable Prototypes in Federated Medical Imaging5 Jul 2026
- 2026FedSPM: Routing-Enabled Federated Learning under Dual Heterogeneity via Semiparametric Mixture5 Jul 2026
- 2026FedXDS: Leveraging Model Attribution Methods to counteract Data Heterogeneity in Federated Learning30 Jun 2026
- 2026FedReLa: Imbalanced Federated Learning via Re-Labeling24 Jun 2026
- 2026TL++: Accuracy and Privacy Preserving Traversal Learning for Distributed Intelligent Systems24 Jun 2026
- 2026HERO: A Heterogeneity-Aware Benchmark Library for Federated Continual Learning13 Jun 2026
- 2026Federated Learning for Feature Generalization with Convex Constraints12 Jun 2026
- 2026FedSPC: Shared Parameter Correction for Personalized Federated Learning11 Jun 2026
- 2026From Data Heterogeneity to Convergence: A Data-Centric Review of Federated Learning9 Jun 2026
- 2026FedEHR-Gen: Federated Synthetic Time-Series EHR Generation via Latent Space Alignment and Distribution-Aware Aggregation27 May 2026
- 2026Pattern Recognition Tasks with Personalized Federated Learning27 May 2026
- 2026A Typed Tensor Language for Shared-State Federated Computation20 May 2026
- 2026OmniISR: A Unified Framework for Centralized and Federated Learning via Intermediate Supervision and Regularization19 May 2026
- 2026Federated Learning by Utility-Constrained Stochastic Aggregation for Improving Rational Participation18 May 2026
- 2026Class-wise Contribution Estimation via Logit Maximization for Federated Learning17 May 2026
- 2026BESplit: Bias-Compensated Split Federated Learning with Evidential Aggregation17 May 2026
- 2026Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning13 May 2026
- 2026FedVSSAM: Mitigating Flatness Incompatibility in Sharpness-Aware Federated Learning9 May 2026
- 2026UMEDA: Unified Multi-modal Efficient Data Fusion for Privacy-Preserving Graph Federated Learning via Spectral-Gated Attention and Diffusion-Based Operator Alignment8 May 2026
- 2026ForgeVLA: Federated Vision-Language-Action Learning without Language Annotations8 May 2026
- 2026FedFrozen: Two-Stage Federated Optimization via Attention Kernel Freezing7 May 2026
- 2026FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings7 May 2026
- 2026Fine-Tuning Impairs the Balancedness of Foundation Models in Long-tailed Personalized Federated Learning4 May 2026
- 2026Personalized Federated Learning for Gradient Alignment4 May 2026
50 newest of 555 All 555 (JSON)
- 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
- 2024FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Modelscited by 17
- 2024Navigating Heterogeneity and Privacy in One-Shot Federated Learning with Diffusion Modelscited by 7
- 2025One-Shot Federated Learning with Classifier-Free Diffusion Modelscited by 7
- 2023One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Modelscited by 5
- 2025One-Shot Federated Learning with Classifier-Free Diffusion Models12 Feb 2025
- 2024FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Models29 Jul 2024
- 2024Navigating Heterogeneity and Privacy in One-Shot Federated Learning with Diffusion Models2 May 2024
- 2023One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models15 Nov 2023
- 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models29 May 2023
- 2024FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learningcited by 6
- 2024F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristicscited by 4
- 2025How (Mis)calibrated is your Federated CLIP and what to do about it?cited by 0
- 2025How (Mis)calibrated is your Federated CLIP and what to do about it?3 Dec 2025
- 2024FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning19 Dec 2024
- 2024F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics17 Nov 2024
- 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
- 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
- 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
- 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
- 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
- 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 108
- 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
- 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
- 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
- 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
- 2021Differentially Private Federated Learning on Heterogeneous Datacited by 74
- 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
12 most cited of 540 All 540 (JSON)
- 2026FedCKA: Representation-Guided Layer Personalization for Federated 3D Perception Across Driving Domains1 Oct 2026
- 2026Latent Information Sharing for Accelerating Federated Learning1 Oct 2026
- 2026A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilization21 Sep 2026
- 2026Joint Domain-Class Modeling for Federated Learning Under Feature Skew19 Sep 2026
- 2026FedeRICo: Federated Region-Influenced Coupling for Traffic Flow Prediction17 Sep 2026
- 2026Structural Negative Transfer in Federated Graph Neural Networks: Diagnosis, Causal Investigation, and the Limits of Divergence-Aware Mitigation15 Sep 2026
- 2026Adaptive Bayesian Partner Selection for Federated Clinical Centers15 Sep 2026
- 2026CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimization14 Sep 2026
- 2026FedDRAW: Federated Dual Reputation Annealing Weighting for Heterogeneous Multi-Institutional Chest Radiograph Classification4 Sep 2026
- 2026FedCMAPSS: A Benchmark for Federated Learning in Remaining Useful Life Estimation26 Aug 2026
- 2026A Momentum-Based Variance-Reduced Algorithm for Federated Multiobjective Optimization24 Aug 2026
- 2026FedADB: Class Anchor-Driven Dual-Branch Federated Learning for Mitigating Forgetting15 Aug 2026
12 newest of 540 All 540 (JSON)