mlsanityData to 14 Oct 2026

Converge Faster, Talk Less: Hessian-Informed Federated Zeroth-Order Optimization

arXiv 2506.02370 ↗First posted 3 Jun 2025cs.LG cs.DCcited by 3

Not on the map. Why?

Abstract (arXiv)

Zeroth-order (ZO) optimization enables dimension-free communication in federated learning (FL), making it attractive for fine-tuning of large language models (LLMs) due to significant communication savings. However, existing ZO-FL methods largely overlook curvature information, despite its well-established benefits for convergence acceleration. To address this, we propose HiSo, a Hessian-informed ZO federated optimization method that accelerates convergence by leveraging global diagonal Hessian approximations, while strictly preserving scalar-only communication without transmitting any second-order information. Theoretically, for non-convex functions, we show that HiSo can achieve an accelerated convergence rate that is independent of the Lipschitz constant L and model dimension d under some Hessian approximation assumptions, offering a plausible explanation for the observed phenomenon of ZO convergence being much faster than its worst-case mathscrO(d)-bound. Empirically, across diverse LLM fine-tuning benchmarks, HiSo delivers a 1~5× speedup in communication rounds over existing state-of-the-art ZO-FL baselines. This superior convergence not only cuts communication costs but also provides strong empirical evidence that Hessian information acts as an effective accelerator in federated ZO optimization settings. Our source code is provided at https://github.com/ZidongLiu/DeComFL.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 36 of its 53 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

36 of 53 references.

References · 36

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2021Convex optimizationcited by 2,515
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  7. 2020Adaptive Federated Optimizationcited by 1,046
  8. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  9. 2023A Comprehensive Overview of Large Language Modelscited by 648
  10. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  11. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  12. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
24 more
  1. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  2. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  3. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  4. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  5. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  6. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  7. 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
  8. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  9. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  10. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 53
  11. 2020Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networkscited by 45
  12. 2021DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learningcited by 41
  13. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 30
  14. 2025A Survey on Federated Fine-tuning of Large Language Modelscited by 24
  15. 2024Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimizationcited by 13
  16. 2023BAFFLE: A Baseline of Backpropagation-Free Federated Learningcited by 12
  17. 2022Communication-Efficient Federated Learning With Data and Client Heterogeneitycited by 9
  18. 2024FedBAT: Communication-Efficient Federated Learning via Learnable Binarizationcited by 9
  19. 2024Adaptive and Optimal Second-order Optimistic Methods for Minimax Optimizationcited by 8
  20. 2021CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learningcited by 7
  21. 2021Curvature-Aware Derivative-Free Optimizationcited by 7
  22. 2022Zeroth-Order Negative Curvature Finding: Escaping Saddle Points without Gradientscited by 6
  23. 2025Exact and Linear Convergence for Federated Learning under Arbitrary Client Participation is Attainablecited by 6
  24. 2024Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspectivecited by 1
  2. 2026Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpoweredcited by 1
  3. 2026HO-FL: Hybrid-Order Federated Learning for Heterogeneous Edge Devicescited by 0