Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning
Not on the map. Why?
Abstract (arXiv)
Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices). However, the data distribution among clients is often non-IID in nature, making efficient optimization difficult. To alleviate this issue, many FL algorithms focus on mitigating the effects of data heterogeneity across clients by introducing a variety of proximal terms, some incurring considerable compute and/or memory overheads, to restrain local updates with respect to the global model. Instead, we consider rethinking solutions to data heterogeneity in FL with a focus on local learning generality rather than proximal restriction. To this end, we first present a systematic study informed by second-order indicators to better understand algorithm effectiveness in FL. Interestingly, we find that standard regularization methods are surprisingly strong performers in mitigating data heterogeneity effects. Based on our findings, we further propose a simple and effective method, FedAlign, to overcome data heterogeneity and the pitfalls of previous methods. FedAlign achieves competitive accuracy with state-of-the-art FL methods across a variety of settings while minimizing computation and memory overhead. Code is available at https://github.com/mmendiet/FedAlign
Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Builds on 31 of its 37 references that are in the map. First posted on arXiv 28 Nov 2021.
Builds on
31 of 37 references.
References · 31
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2016Deep Networks with Stochastic Depthcited by 973
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
- 2021Model-Contrastive Federated Learningcited by 762
- 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
19 more
- 2020Federated Learning with Matched Averagingcited by 564
- 2021Federated Learning Based on Dynamic Regularizationcited by 555
- 2019Fantastic Generalization Measures and Where to Find Themcited by 507
- 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
- 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
- 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
- 2018DropBlock: A regularization method for convolutional networkscited by 261
- 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
- 2014Efficient Object Localization Using Convolutional Networkscited by 226
- 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
- 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
- 2020Learning explanations that are hard to varycited by 132
- 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
- 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
- 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
- 2020XOR Mixup: Privacy-Preserving Data Augmentation for One-Shot Federated Learningcited by 58
- 2020Mix2FLD: Downlink Federated Learning After Uplink Federated Distillation With Two-Way Mixupcited by 29
- 2021Lipschitz Continuity Guided Knowledge Distillationcited by 19
- 2020GradAug: A New Regularization Method for Deep Neural Networkscited by 18
Built on by
Citing papers per year: 2022: 9; 2023: 25; 2024: 39; 2025: 32; 2026: 9.
114 papers.
Sort:
By area:
- 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
- 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
- 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 59
- 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
- 2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 39
- 2022Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspacescited by 36
- 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
- 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
- 2023L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learningcited by 29
- 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillationcited by 25
- 2022FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentationcited by 24
- 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 23
38 more
- 2023Federated Learning for Computer Visioncited by 22
- 2023FedPerfix: Towards Partial Model Personalization of Vision Transformers in Federated Learningcited by 21
- 2024Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modalitycited by 21
- 2023Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrastcited by 17
- 2023FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learningcited by 14
- 2023Exploiting Label Skews in Federated Learning with Model Concatenationcited by 14
- 2024Federated Distillation: A Surveycited by 14
- 2022PGFed: Personalize Each Client's Global Objective for Federated Learningcited by 13
- 2023Multimodal Federated Learning in Healthcare: a Reviewcited by 13
- 2024FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneitycited by 13
- 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
- 2024Towards Multi-modal Transformers in Federated Learningcited by 12
- 2024Federated Online Adaptation for Deep Stereocited by 12
- 2022Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learningcited by 11
- 2025FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Modelscited by 11
- 2025Geometric Knowledge-Guided Localized Global Distribution Alignment for Federated Learningcited by 11
- 2023Federated Learning Across Decentralized and Unshared Archives for Remote Sensing Image Classificationcited by 10
- 2023pFedSim: Similarity-Aware Model Aggregation Towards Personalized Federated Learningcited by 9
- 2024FedUV: Uniformity and Variance for Heterogeneous Federated Learningcited by 8
- 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 8
- 2024Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Driftcited by 8
- 2022SemiPFL: Personalized Semi-Supervised Federated Learning Framework for Edge Intelligencecited by 7
- 2023Towards Instance-adaptive Inference for Federated Learningcited by 7
- 2024Navigating Heterogeneity and Privacy in One-Shot Federated Learning with Diffusion Modelscited by 7
- 2024Beyond the Federation: Topology-aware Federated Learning for Generalization to Unseen Clientscited by 7
- 2025Federated Learning with Domain Shift Erasercited by 7
- 2023Memory-adaptive Depth-wise Heterogeneous Federated Learningcited by 5
- 2025A Thorough Assessment of the Non-IID Data Impact in Federated Learningcited by 5
- 2023Federated Model Aggregation via Self-Supervised Priors for Highly Imbalanced Medical Image Classificationcited by 4
- 2024Transformer-based Federated Learning for Multi-Label Remote Sensing Image Classificationcited by 4
- 2024FedSAC: Dynamic Submodel Allocation for Collaborative Fairness in Federated Learningcited by 4
- 2024Federated Learning Client Pruning for Noisy Labelscited by 4
- 2024FedCFA: Alleviating Simpson's Paradox in Model Aggregation with Counterfactual Federated Learningcited by 4
- 2025Interaction-Aware Gaussian Weighting for Clustered Federated Learningcited by 4
- 2025FedCanon: Non-Convex Composite Federated Learning with Efficient Proximal Operation on Heterogeneous Datacited by 4
- 2022Dual Class-Aware Contrastive Federated Semi-Supervised Learningcited by 3
- 2023Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectivescited by 3
- 2023Rethinking Client Drift in Federated Learning: A Logit Perspectivecited by 3
50 most cited of 114 All 114 (JSON)
- 2026FedXDS: Leveraging Model Attribution Methods to counteract Data Heterogeneity in Federated Learning30 Jun 2026
- 2026Federated Learning for Feature Generalization with Convex Constraints12 Jun 2026
- 2026Multi-Level Analyzation of Imbalance to Resolve Non-IID-Ness in Federated Learning8 Jun 2026
- 2026Q-LocalAdam: Memory-Efficient Client-Side Adaptive Optimization for Edge Federated Learning17 May 2026
- 2026FedHPro: Federated Hyper-Prototype Learning via Gradient Matching13 May 2026
- 2026FedOUI: OUI-Guided Client Weighting for Federated Aggregation12 May 2026
- 2026Enhancing Federated Quadruplet Learning: Stochastic Client Selection and Embedding Stability Analysis8 May 2026
- 2026Optimizing Multilingual LLMs via Federated Learning: A Study of Client Language Composition25 Mar 2026
- 2026FedRG: Unleashing the Representation Geometry for Federated Learning with Noisy Clients20 Mar 2026
- 2025FedDPC : Handling Data Heterogeneity and Partial Client Participation in Federated Learning23 Dec 2025
- 2025Minimizing Layerwise Activation Norm Improves Generalization in Federated Learning9 Dec 2025
- 2025CO-PFL: Contribution-Oriented Personalized Federated Learning for Heterogeneous Networks23 Oct 2025
38 more
- 2025CEPerFed: Communication-Efficient Personalized Federated Learning for Multi-Pulse MRI Classification20 Oct 2025
- 2025C^2Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learning24 Sep 2025
- 2025On the Out-of-Distribution Backdoor Attack for Federated Learning16 Sep 2025
- 2025FedRP: A Communication-Efficient Approach for Differentially Private Federated Learning Using Random Projection12 Sep 2025
- 2025FedQuad: Federated Stochastic Quadruplet Learning to Mitigate Data Heterogeneity4 Sep 2025
- 2025Generalizable Federated Learning using Client Adaptive Focal Modulation14 Aug 2025
- 2025Decoupled Contrastive Learning for Federated Learning6 Aug 2025
- 2025FairFedMed: Benchmarking Group Fairness in Federated Medical Imaging with FairLoRA21 Jul 2025
- 2025Federated Reinforcement Learning in Heterogeneous Environments19 Jul 2025
- 2025FedDifRC: Unlocking the Potential of Text-to-Image Diffusion Models in Heterogeneous Federated Learning9 Jul 2025
- 2025UniVarFL: Uniformity and Variance Regularized Federated Learning for Heterogeneous Data9 Jun 2025
- 2025Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning28 May 2025
- 2025Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments26 May 2025
- 2025ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoning22 May 2025
- 2025How to Backdoor Image Knowledge Distillation30 Apr 2025
- 2025FedCanon: Non-Convex Composite Federated Learning with Efficient Proximal Operation on Heterogeneous Data16 Apr 2025
- 2025A Thorough Assessment of the Non-IID Data Impact in Federated Learning21 Mar 2025
- 2025Redefining non-IID Data in Federated Learning for Computer Vision Tasks: Migrating from Labels to Embeddings for Task-Specific Data Distributions17 Mar 2025
- 2025GC-Fed: Gradient Centralized Federated Learning with Partial Client Participation17 Mar 2025
- 2025Federated Learning with Domain Shift Eraser17 Mar 2025
- 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRA14 Mar 2025
- 2025PRISM: Privacy-Preserving Improved Stochastic Masking for Federated Generative Models11 Mar 2025
- 2025HFedCKD: Toward Robust Heterogeneous Federated Learning via Data-free Knowledge Distillation and Two-way Contrast9 Mar 2025
- 2025Geometric Knowledge-Guided Localized Global Distribution Alignment for Federated Learning9 Mar 2025
- 2025FinP: Fairness-in-Privacy in Federated Learning by Addressing Disparities in Privacy Risk25 Feb 2025
- 2025Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead10 Feb 2025
- 2025Interaction-Aware Gaussian Weighting for Clustered Federated Learning5 Feb 2025
- 2025Mitigating Domain Shift in Federated Learning via Intra- and Inter-Domain Prototypes15 Jan 2025
- 2025FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models5 Jan 2025
- 2024Asymmetrical Reciprocity-based Federated Learning for Resolving Disparities in Medical Diagnosis27 Dec 2024
- 2024FedCFA: Alleviating Simpson's Paradox in Model Aggregation with Counterfactual Federated Learning25 Dec 2024
- 2024FedTLU: Federated Learning with Targeted Layer Updates23 Dec 2024
- 2024FedGAT: A Privacy-Preserving Federated Approximation Algorithm for Graph Attention Networks20 Dec 2024
- 2024Hybrid-Regularized Magnitude Pruning for Robust Federated Learning under Covariate Shift19 Dec 2024
- 2024Non-Convex Optimization in Federated Learning via Variance Reduction and Adaptive Learning16 Dec 2024
- 2024Mind the Cost of Scaffold! Benign Clients May Even Become Accomplices of Backdoor Attack25 Nov 2024
- 2024Federated Learning Client Pruning for Noisy Labels11 Nov 2024
- 2024PARDON: Privacy-Aware and Robust Federated Domain Generalization30 Oct 2024
50 newest of 114 All 114 (JSON)
- 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
- 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
- 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 59
- 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
- 2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 39
- 2022Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspacescited by 36
- 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
- 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
- 2023L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learningcited by 29
- 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillationcited by 25
- 2022FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentationcited by 24
- 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 23
12 most cited of 108 All 108 (JSON)
- 2026FedXDS: Leveraging Model Attribution Methods to counteract Data Heterogeneity in Federated Learning30 Jun 2026
- 2026Federated Learning for Feature Generalization with Convex Constraints12 Jun 2026
- 2026Multi-Level Analyzation of Imbalance to Resolve Non-IID-Ness in Federated Learning8 Jun 2026
- 2026Q-LocalAdam: Memory-Efficient Client-Side Adaptive Optimization for Edge Federated Learning17 May 2026
- 2026FedHPro: Federated Hyper-Prototype Learning via Gradient Matching13 May 2026
- 2026FedOUI: OUI-Guided Client Weighting for Federated Aggregation12 May 2026
- 2026Enhancing Federated Quadruplet Learning: Stochastic Client Selection and Embedding Stability Analysis8 May 2026
- 2026Optimizing Multilingual LLMs via Federated Learning: A Study of Client Language Composition25 Mar 2026
- 2026FedRG: Unleashing the Representation Geometry for Federated Learning with Noisy Clients20 Mar 2026
- 2025FedDPC : Handling Data Heterogeneity and Partial Client Participation in Federated Learning23 Dec 2025
- 2025Minimizing Layerwise Activation Norm Improves Generalization in Federated Learning9 Dec 2025
- 2025CO-PFL: Contribution-Oriented Personalized Federated Learning for Heterogeneous Networks23 Oct 2025
12 newest of 108 All 108 (JSON)