# Fed-Sophia: A Communication-Efficient Second-Order Federated Learning Algorithm

arXiv: https://arxiv.org/abs/2406.06655 · HTML: https://mlsanity.com/paper/2406.06655/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training](https://mlsanity.com/paper/2305.14342/) (2023)
- [ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning](https://mlsanity.com/paper/2006.00719/) (2020)
- [Federated Learning with Compression: Unified Analysis and Sharp Guarantees](https://mlsanity.com/paper/2007.01154/) (2020)
- [FedNL: Making Newton-Type Methods Applicable to Federated Learning](https://mlsanity.com/paper/2106.02969/) (2021)
- [FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learning](https://mlsanity.com/paper/2206.08829/) (2022)
- [DONE: Distributed Approximate Newton-type Method for Federated Edge Learning](https://mlsanity.com/paper/2012.05625/) (2020)
- [Partial Model Averaging in Federated Learning: Performance Guarantees and Benefits](https://mlsanity.com/paper/2201.03789/) (2022)
- [FLECS: A Federated Learning Second-Order Framework via Compression and Sketching](https://mlsanity.com/paper/2206.02009/) (2022)

## Built on by (most cited first, 5 of 5)

- [Taming Preconditioner Drift: Unlocking the Potential of Second-Order Optimizers for Federated Learning on Non-IID Data](https://mlsanity.com/paper/2602.19271/) (2026)
- [Scalable and Resource-Efficient Second-Order Federated Learning via Over-the-Air Aggregation](https://mlsanity.com/paper/2410.07662/) (2024)
- [Analytic Personalized Federated Meta-Learning](https://mlsanity.com/paper/2502.06915/) (2025)
- [Accelerated Training of Federated Learning via Second-Order Methods](https://mlsanity.com/paper/2505.23588/) (2025)
- [FedMIX-P: Mixing Local and Global Preconditioners for Federated Vision and Language Model Training](https://mlsanity.com/paper/2610.01515/) (2026)

## Abstract (arXiv)

Federated learning is a machine learning approach where multiple devices collaboratively learn with the help of a parameter server by sharing only their local updates. While gradient-based optimization techniques are widely adopted in this domain, the curvature information that second-order methods exhibit is crucial to guide and speed up the convergence. This paper introduces a scalable second-order method, allowing the adoption of curvature information in federated large models. Our method, coined Fed-Sophia, combines a weighted moving average of the gradient with a clipping operation to find the descent direction. In addition to that, a lightweight estimation of the Hessian's diagonal is used to incorporate the curvature information. Numerical evaluation shows the superiority, robustness, and scalability of the proposed Fed-Sophia scheme compared to first and second-order baselines.