Subspace Optimization for Efficient Federated Learning under Heterogeneous Data
Not on the map. Why?
Abstract (arXiv)
Federated learning increasingly operates in a large-model regime where communication, memory, and computation are all scarce. Typically, non-IID client data induce drift that degrades the stability and performance of local training. Existing remedies such as SCAFFOLD introduce heterogeneity-correction mechanisms to address this challenge, but they incur substantial extra communication and memory overhead. This paper proposes a subspace optimization method for federated learning (SSF), which performs heterogeneity-corrected optimization in a low-dimensional subspace using only projected quantities, while preserving full-dimensional control information through a backfill-style update that retains residual components whenever the active subspace changes. Under standard smoothness and bounded-variance assumptions, SSF attains a non-asymptotic rate of order widetildeO(1/T+1/√NKT). Experiments show favorable accuracy--efficiency trade-offs under heterogeneous data.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 34 references that are in the map. First posted on arXiv 28 Apr 2026.
Builds on
30 of 34 references.
References · 30
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
- 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
- 2018Local SGD Converges Fast and Communicates Littlecited by 595
- 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
- 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
- 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
- 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 179
- 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100
18 more
- 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 98
- 2020Federated Accelerated Stochastic Gradient Descentcited by 85
- 2022An Improved Analysis of Gradient Tracking for Decentralized Machine Learningcited by 68
- 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 35
- 2023Decentralized Gradient Tracking with Local Stepscited by 32
- 2023Momentum Provably Improves Error Feedback!cited by 31
- 2020Heterogeneous Federated Learningcited by 19
- 2025FedMuon: Accelerating Federated Learning with Matrix Orthogonalizationcited by 15
- 2022Fast Composite Optimization and Statistical Recovery in Federated Learningcited by 12
- 2023GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Trainingcited by 10
- 2025Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guaranteescited by 9
- 2024Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learningcited by 8
- 2023Towards Bias Correction of FedAvg over Nonuniform and Time-Varying Communicationscited by 6
- 2020A Unified Linear Speedup Analysis of Federated Averaging and Nesterov FedAvgcited by 4
- 2025Non-convex composite federated learning with heterogeneous datacited by 4
- 2025FedCanon: Non-Convex Composite Federated Learning with Efficient Proximal Operation on Heterogeneous Datacited by 4
- 2025An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Datacited by 4
- 2025Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvgcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.