mlsanityData to 14 Oct 2026

Personalized Federated Learning for Gradient Alignment

arXiv 2605.02143 ↗First posted 4 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Personalized federated learning (pFL) aims to adapt models to client specific data distributions, yet it often fails to reliably preserve personalized information. Local training is hindered by high variance gradients induced by limited and heterogeneous client data, while aggregation further distorts client specific optimization directions. To address these challenges, we propose pFLAlign, a gradient alignment framework to maintain client specific information during both local training and aggregation. pFLAlign consists of two complementary mechanisms: one adapts local gradient directions to reduce variance during client side optimization, and the other mitigates aggregation induced distortion by realigning the global model with each client's personalized direction. Theoretically, we derive pFLAlign from a PAC Bayesian analysis, which reveals how personalized gradient alignment preserves client specific information. Our experiments and ablation studies show that pFLAlign consistently improves personalization performance and training stability, achieving state of the art results.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 32 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

25 of 32 references.

References · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  7. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  8. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  9. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  10. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  11. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  12. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
13 more
  1. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  2. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  3. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  4. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  5. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  6. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  7. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
  8. 2025In Search of Adam's Secret Saucecited by 40
  9. 2022Online PAC-Bayes Learningcited by 22
  10. 2025FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Modelscited by 15
  11. 2025FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRAcited by 11
  12. 2024Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learningcited by 8
  13. 2023A Coefficient Makes SVRG Effectivecited by 7

Built on by

Not cited yet.