mlsanityData to 14 Oct 2026

Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization

arXiv 2405.15861 ↗First posted 24 May 2024cs.LG cs.DCcited by 13

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) offers a promising framework for collaborative and privacy-preserving machine learning across distributed data sources. However, the substantial communication costs associated with FL significantly challenge its efficiency. Specifically, in each communication round, the communication costs scale linearly with the model's dimension, which presents a formidable obstacle, especially in large model scenarios. Despite various communication-efficient strategies, the intrinsic dimension-dependent communication cost remains a major bottleneck for current FL implementations. This paper proposes a novel dimension-free communication algorithm - DeComFL, which leverages the zeroth-order optimization techniques and reduces the communication cost from mathscrO(d) to mathscrO(1) by transmitting only a constant number of scalar values between clients and the server in each round, regardless of the dimension d of the model parameters. Theoretically, in non-convex functions, we prove that our algorithm achieves state-of-the-art rates, which show a linear speedup of the number of clients and local steps under standard assumptions. With additional low effective rank assumption, we can further show the convergence rate is independent of the model dimension d as well. Empirical evaluations, encompassing both classic deep learning training and large language model fine-tuning, demonstrate significant reductions in communication overhead. Notably, DeComFL achieves this by transmitting only around 1MB of data in total between the server and a client to fine-tune a model with billions of parameters. Our code is available at https://github.com/ZidongLiu/DeComFL.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 57 of its 81 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

57 of 81 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2019Advances and Open Problems in Federated Learningcited by 3,046
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,847
38 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  2. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  3. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
  4. 2018Rethinking the Value of Network Pruningcited by 599
  5. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  6. 2018Sparsified SGD with Memorycited by 376
  7. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  8. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  9. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  10. 2021A Field Guide to Federated Optimizationcited by 315
  11. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  12. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  13. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  14. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  15. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 198
  16. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 162
  17. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 160
  18. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  19. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 111
  20. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  21. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  22. 2020Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approachcited by 81
  23. 2019ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimizationcited by 76
  24. 2021Federated Learning Meets Blockchain in Edge Computing: Opportunities and Challengescited by 76
  25. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  26. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuningcited by 64
  27. 2021Adaptive Quantization of Model Updates for Communication-Efficient Federated Learningcited by 63
  28. 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
  29. 2019Understanding Top-k Sparsification in Distributed Deep Learningcited by 56
  30. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  31. 2021A Zeroth-Order Block Coordinate Descent Algorithm for Huge-Scale Black-Box Optimizationcited by 46
  32. 2020Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networkscited by 45
  33. 2021DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learningcited by 41
  34. 2020Communication-Efficient Federated Learning via Optimal Client Samplingcited by 39
  35. 2019Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approachcited by 38
  36. 2020A Review of Winograd Schema Challenge Datasets and Approachescited by 27
  37. 2021Time-Correlated Sparsification for Communication-Efficient Federated Learningcited by 26
  38. 2024On the Convergence of Zeroth-Order Federated Tuning for Large Language Modelscited by 19

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

  1. 2025Towards Straggler-Resilient Split Federated Learning: An Unbalanced Update Approachcited by 7
  2. 2025Converge Faster, Talk Less: Hessian-Informed Federated Zeroth-Order Optimizationcited by 3
  3. 2026HOSL: Hybrid-Order Split Learning for Memory-Constrained Edge Trainingcited by 3
  4. 2026Lean Clients, Full Accuracy: Hybrid Zeroth- and First-Order Split Federated Learningcited by 2
  5. 2026SeedFlood: A Step Toward Scalable Decentralized Fine-Tuning of LLMscited by 1
  6. 2026HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregationcited by 1
  7. 2026Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspectivecited by 1
  8. 2026Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpoweredcited by 1
  9. 2025Efficient Zeroth-Order Federated Finetuning of Language Models on Resource-Constrained Devicescited by 0
  10. 2025FedSGT: Exact Federated Unlearning via Sequential Group-based Trainingcited by 0
  11. 2026ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activationcited by 0
  12. 2026SPADE-DFL: Communication-Efficient Decentralized Federated Learning via Derivative-Free Linearized ADMMcited by 0
1 more
  1. 2026HO-FL: Hybrid-Order Federated Learning for Heterogeneous Edge Devicescited by 0