mlsanityData to 7 Oct 2026

Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignment

arXiv 2602.01746 ↗First posted 2 Feb 2026cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) is widely used for federated fine-tuning. Yet under non-IID settings, it can substantially underperform full-parameter fine-tuning. Through with-high-probability robustness analysis, we uncover that this gap can be attributed to two coupled mismatches: (i) update-space mismatch, where clients optimize in a low-rank subspace but aggregation occurs in the full space; and (ii) optimizer-state mismatch, where unsynchronized adaptive states amplify drift across rounds. We propose FedGaLore, which combines client-side GaLore-style gradient-subspace optimization with server-side drift-robust synchronization of projected second-moment states via spectral shared-signal extraction, to address this challenge. Across NLU, vision, and NLG benchmarks, FedGaLore improves robustness and accuracy over state-of-the-art federated LoRA baselines in non-IID settings.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 55 of its 70 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

55 of 70 references.

References · 55

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2023Segment Anythingcited by 8,158
  9. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  3. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  4. 2020Federated Learning with Matched Averagingcited by 564
  5. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  6. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  7. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  8. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  9. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  10. 2021A Field Guide to Federated Optimizationcited by 312
  11. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  12. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  13. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  14. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  15. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  16. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  17. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
  18. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  19. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  20. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
  21. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  22. 2020Practical One-Shot Federated Learning for Cross-Silo Settingcited by 76
  23. 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
  24. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  25. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 37
  26. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
  27. 2022Faster Adaptive Federated Learningcited by 35
  28. 2024Subspace Optimization for Large Language Models with Convergence Guaranteescited by 31
  29. 2017Angle-Based Joint and Individual Variation Explainedcited by 29
  30. 2023Proving Linear Mode Connectivity of Neural Networks via Optimal Transportcited by 24
  31. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  32. 2025A Survey on Federated Fine-tuning of Large Language Modelscited by 21
  33. 2022Personalized PCA: Decoupling Shared and Unique Featurescited by 17
  34. 2025FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Modelscited by 15
  35. 2020Latent space models for multiplex networks with shared structurecited by 14
  36. 2025GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projectioncited by 13
  37. 2024Convergence of Distributed Adaptive Optimization with Local Updatescited by 10
  38. 2020mvlearn: Multiview Machine Learning in Pythoncited by 7

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
  2. 2026FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptationcited by 0