mlsanityData to 14 Oct 2026

An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Data

arXiv 2509.05213 ↗First posted 5 Sep 2025cs.LG cs.DCcited by 4

Not on the map. Why?

Abstract (arXiv)

This work addresses the key challenges of applying federated learning to large-scale deep neural networks, particularly the issue of client drift due to data heterogeneity across clients and the high costs of communication, computation, and memory. We propose FedSub, an efficient subspace algorithm for federated learning on heterogeneous data. Specifically, FedSub utilizes subspace projection to guarantee local updates of each client within low-dimensional subspaces, thereby reducing communication, computation, and memory costs. Additionally, it incorporates low-dimensional dual variables to mitigate client drift. We provide convergence analysis that reveals the impact of key factors such as step size and subspace projection matrices on convergence. Experimental results demonstrate its efficiency.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 43 of its 56 references that are in the map. First posted on arXiv 5 Sep 2025.

Builds on

43 of 56 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  7. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  8. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  9. 2021Towards Personalized Federated Learningcited by 464
  10. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  11. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  12. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
31 more
  1. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  2. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  3. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 145
  4. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  5. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  6. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  7. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  8. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  9. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  10. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  11. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 82
  12. 2024Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Modelscited by 54
  13. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 53
  14. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 50
  15. 2021A Unified and Refined Convergence Analysis for Non-Convex Decentralized Learningcited by 44
  16. 2023Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learningcited by 41
  17. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
  18. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 38
  19. 2024Subspace Optimization for Large Language Models with Convergence Guaranteescited by 34
  20. 2023Why Batch Normalization Damage Federated Learning on Non-IID Data?cited by 29
  21. 2023SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Modelscited by 28
  22. 2021Zeroth order optimization with orthogonal random directionscited by 21
  23. 2023z-SignFedAvg: A Unified Stochastic Sign-based Compression for Federated Learningcited by 17
  24. 2025A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Modelscited by 16
  25. 2025FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRAcited by 11
  26. 2023Federated Learning with Manifold Regularization and Normalized Update Reaggregationcited by 8
  27. 2024Communication-Efficient Personalized Federated Learning for Speech-to-Text Taskscited by 8
  28. 2019Compressed Gradient Methods with Hessian-Aided Error Compensationcited by 6
  29. 2024FedLion: Faster Adaptive Federated Optimization with Fewer Communicationcited by 6
  30. 2024Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularizationcited by 6
  31. 2024Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projectioncited by 1
  2. 2026Subspace Optimization for Efficient Federated Learning under Heterogeneous Datacited by 1
  3. 2026ReasFlow: Assisting Reasoning-Centric Scientific Discovery in Applied Mathematics via a Knowledge-Based Multi-Agent Systemcited by 1
  4. 2026Exploiting Correlations in Federated Learning: Opportunities and Practical Limitationscited by 0