mlsanityData to 7 Oct 2026

Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning

arXiv 2201.09531 ↗First posted 24 Jan 2022cs.LG cs.AI eess.SPcited by 59

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL), as an emerging edge artificial intelligence paradigm, enables many edge devices to collaboratively train a global model without sharing their private data. To enhance the training efficiency of FL, various algorithms have been proposed, ranging from first-order to second-order methods. However, these algorithms cannot be applied in scenarios where the gradient information is not available, e.g., federated black-box attack and federated hyperparameter tuning. To address this issue, in this paper we propose a derivative-free federated zeroth-order optimization (FedZO) algorithm featured by performing multiple local updates based on stochastic gradient estimators in each communication round and enabling partial device participation. Under non-convex settings, we derive the convergence performance of the FedZO algorithm on non-independent and identically distributed data and characterize the impact of the numbers of local iterates and participating edge devices on the convergence. To enable communication-efficient FedZO over wireless networks, we further propose an over-the-air computation (AirComp) assisted FedZO algorithm. With an appropriate transceiver design, we show that the convergence of AirComp-assisted FedZO can still be preserved under certain signal-to-noise ratio conditions. Simulation results demonstrate the effectiveness of the FedZO algorithm and validate the theoretical observations.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 38 of its 53 references that are in the map. First posted on arXiv 24 Jan 2022.

Builds on

38 of 53 references.

References · 38

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  7. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  8. 2021A Field Guide to Federated Optimizationcited by 312
  9. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  10. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  11. 2019A Joint Learning and Communications Framework for Federated Learning over Wireless Networkscited by 262
  12. 2018Federated Learning via Over-the-Air Computationcited by 215
26 more
  1. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 190
  2. 2018Broadband Analog Aggregation for Low-Latency Federated Edge Learning (Extended Version)cited by 186
  3. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 160
  4. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 143
  5. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 109
  6. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  7. 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
  8. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 91
  9. 2020Federated Bayesian Optimization via Thompson Samplingcited by 77
  10. 2020FedDANE: A Federated Newton-Type Methodcited by 76
  11. 2020Over-the-Air Federated Learning from Heterogeneous Datacited by 72
  12. 2020Scheduling for Cellular Federated Edge Learning with Importance and Channel Awarenesscited by 62
  13. 2021Optimized Power Control Design for Over-the-Air Federated Edge Learningcited by 45
  14. 2017GIANT: Globally Improved Approximate Newton Method for Distributed Optimizationcited by 43
  15. 2020Federated Learning via Intelligent Reflecting Surfacecited by 38
  16. 2020Reconfigurable Intelligent Surface Enabled Federated Learning: A Unified Communication-Learning Design Approachcited by 32
  17. 2017Zeroth-Order Online Alternating Direction Method of Multipliers: Convergence Analysis and Applicationscited by 22
  18. 2020Federated Machine Learning for Intelligent IoT via Reconfigurable Intelligent Surfacecited by 21
  19. 2021Accelerated Gradient Descent Learning over Multiple Access Fading Channelscited by 14
  20. 2017Zeroth Order Nonconvex Multi-Agent Optimization over Networkscited by 13
  21. 2021Zeroth-Order Algorithms for Stochastic Distributed Nonconvex Optimizationcited by 13
  22. 2021Over-the-Air Computation via Reconfigurable Intelligent Surfacecited by 12
  23. 2019Time Synchronization in 5G Wireless Edge: Requirements and Solutions for Critical-MTCcited by 7
  24. 2021UAV Aided Over-the-Air Computationcited by 5
  25. 2018Distributed Zeroth Order Optimization Over Random Networks: A Kiefer-Wolfowitz Stochastic Approximation Approachcited by 3
  26. 2021Distributed Zeroth-Order Stochastic Optimization in Time-varying Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 14; 2024: 14; 2025: 16; 2026: 12.

59 papers.

Sort:

  1. 2023Decentralized Federated Learning: A Survey and Perspectivecited by 86
  2. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  3. 2023Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challengescited by 39
  4. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  5. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  6. 2022A Survey on Over-the-Air Computationcited by 19
  7. 2024On the Convergence of Zeroth-Order Federated Tuning for Large Language Modelscited by 18
  8. 2022Interference Management for Over-the-Air Federated Learning in Multi-Cell Wireless Networkscited by 15
  9. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  10. 2025Can Textual Gradient Work in Federated Learning?cited by 14
  11. 2023BAFFLE: A Baseline of Backpropagation-Free Federated Learningcited by 12
  12. 2024Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimizationcited by 12
38 more
  1. 2023Over-the-Air Federated Learning and Optimizationcited by 10
  2. 2024Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selectioncited by 10
  3. 2023ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learningcited by 9
  4. 2023Federated Zeroth-Order Optimization using Trajectory-Informed Surrogate Gradientscited by 8
  5. 2024Hierarchical Federated Learning with Multi-Timescale Gradient Correctioncited by 7
  6. 2024Fully Zeroth-Order Bilevel Programming via Gaussian Smoothingcited by 6
  7. 2024Ferret: Federated Full-Parameter Tuning at Scale for Large Language Modelscited by 6
  8. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  9. 2023FedZeN: Towards superlinear zeroth-order federated learning via incremental Hessian estimationcited by 4
  10. 2023FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communicationcited by 4
  11. 2024Rendering Wireless Environments Useful for Gradient Estimators: A Zero-Order Stochastic Federated Learning Methodcited by 4
  12. 2024Communication-Efficient Byzantine-Resilient Federated Zero-Order Optimizationcited by 4
  13. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  14. 2025Age-Based Device Selection and Transmit Power Optimization in Over-the-Air Federated Learningcited by 3
  15. 2025Cooperative Decentralized Backdoor Attacks on Vertical Federated Learningcited by 3
  16. 2025Converge Faster, Talk Less: Hessian-Informed Federated Zeroth-Order Optimizationcited by 3
  17. 2026HOSL: Hybrid-Order Split Learning for Memory-Constrained Edge Trainingcited by 3
  18. 2022Lazy Queries Can Reduce Variance in Zeroth-order Optimizationcited by 2
  19. 2025Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMscited by 2
  20. 2025TAP: Two-Stage Adaptive Personalization of Multi-Task and Multi-Modal Foundation Models in Federated Learningcited by 2
  21. 2026Lean Clients, Full Accuracy: Hybrid Zeroth- and First-Order Split Federated Learningcited by 2
  22. 2026FedKRSO: Communication and Memory Efficient Federated Fine-Tuning of Large Language Modelscited by 2
  23. 2023Digital Over-the-Air Federated Learning in Multi-Antenna Systemscited by 1
  24. 2023Federated Learning over Hierarchical Wireless Networks: Training Latency Minimization via Submodel Partitioningcited by 1
  25. 2024Heterogeneous Federated Learning with Splited Language Modelcited by 1
  26. 2024Hierarchical Learning and Computing over Space-Ground Integrated Networkscited by 1
  27. 2025Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Samplingcited by 1
  28. 2025Federated Learning on Riemannian Manifolds: A Gradient-Free Projection-Based Approachcited by 1
  29. 2025Communication-Efficient Zero-Order and First-Order Federated Learning Methods over Wireless Networkscited by 1
  30. 2025More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learningcited by 1
  31. 2026HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregationcited by 1
  32. 2026Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpoweredcited by 1
  33. 2023Communication-Efficient Zeroth-Order Distributed Online Optimization: Algorithm, Theory, and Applicationscited by 0
  34. 2023Communication Efficient and Privacy-Preserving Federated Learning Based on Evolution Strategiescited by 0
  35. 2024A Historical Trajectory Assisted Optimization Method for Zeroth-Order Federated Learningcited by 0
  36. 2025Analytic Personalized Federated Meta-Learningcited by 0
  37. 2025Efficient Zeroth-Order Federated Finetuning of Language Models on Resource-Constrained Devicescited by 0
  38. 2025Communication-Efficient and Differentially Private Vertical Federated Learning with Zeroth-Order Optimizationcited by 0

50 most cited of 59 All 59 (JSON)