# Faster On-Device Training Using New Federated Momentum Algorithm

arXiv: https://arxiv.org/abs/2002.02090 · HTML: https://mlsanity.com/paper/2002.02090/

## Key facts

- Cited by 25 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 6 Feb 2020.

## Built on by (most cited first, 25 of 25)

- [Personalized Federated Learning using Hypernetworks](https://mlsanity.com/paper/2103.04628/) (2021)
- [DiLoCo: Distributed Low-Communication Training of Language Models](https://mlsanity.com/paper/2311.08105/) (2023)
- [FedCM: Federated Learning with Client-level Momentum](https://mlsanity.com/paper/2106.10874/) (2021)
- [The Future of Large Language Model Pre-training is Federated](https://mlsanity.com/paper/2405.10853/) (2024)
- [Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch](https://mlsanity.com/paper/2501.18512/) (2025)
- [Asynchronous Local-SGD Training for Language Modeling](https://mlsanity.com/paper/2401.09135/) (2024)
- [Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer](https://mlsanity.com/paper/2109.08119/) (2021)
- [Eager Updates For Overlapped Communication and Computation in DiLoCo](https://mlsanity.com/paper/2502.12996/) (2025)
- [Coordinating Momenta for Cross-silo Federated Learning](https://mlsanity.com/paper/2102.03970/) (2021)
- [Federated Optimization of Smooth Loss Functions](https://mlsanity.com/paper/2201.01954/) (2022)
- [Hierarchical Federated Learning with Momentum Acceleration in Multi-Tier Networks](https://mlsanity.com/paper/2210.14560/) (2022)
- [FilFL: Client Filtering for Optimized Client Participation in Federated Learning](https://mlsanity.com/paper/2302.06599/) (2023)
- [Worldwide Federated Training of Language Models](https://mlsanity.com/paper/2405.14446/) (2024)
- [Review of Mathematical Optimization in Federated Learning](https://mlsanity.com/paper/2412.01630/) (2024)
- [An Operator Splitting View of Federated Learning](https://mlsanity.com/paper/2108.05974/) (2021)
- [FEDSTR: Money-In AI-Out | A Decentralized Marketplace for Federated Learning and LLM Training on the NOSTR Protocol](https://mlsanity.com/paper/2404.15834/) (2024)
- [MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates](https://mlsanity.com/paper/2510.05361/) (2025)
- [FedCos: A Scene-adaptive Federated Optimization Enhancement for Performance Improvement](https://mlsanity.com/paper/2204.03174/) (2022)
- [Semi-Variance Reduction for Fair Federated Learning](https://mlsanity.com/paper/2406.16193/) (2024)
- [OmniFed: A Modular Framework for Configurable Federated Learning from Edge to HPC](https://mlsanity.com/paper/2509.19396/) (2025)
- [Federated Stochastic Gradient Descent Begets Self-Induced Momentum](https://mlsanity.com/paper/2202.08402/) (2022)
- [Federated Learning for Energy Constrained IoT devices: A systematic mapping study](https://mlsanity.com/paper/2301.03720/) (2023)
- [Photon: Federated LLM Pre-Training](https://mlsanity.com/paper/2411.02908/) (2024)
- [A Momentum-Based Variance-Reduced Algorithm for Federated Multiobjective Optimization](https://mlsanity.com/paper/2608.22945/) (2026)
- [Accelerating Sharded Data Parallelism at Scale with Federated Learning](https://mlsanity.com/paper/2609.20359/) (2026)

## Abstract (arXiv)

Mobile crowdsensing has gained significant attention in recent years and has become a critical paradigm for emerging Internet of Things applications. The sensing devices continuously generate a significant quantity of data, which provide tremendous opportunities to develop innovative intelligent applications. To utilize these data to train machine learning models while not compromising user privacy, federated learning has become a promising solution. However, there is little understanding of whether federated learning algorithms are guaranteed to converge. We reconsider model averaging in federated learning and formulate it as a gradient-based method with biased gradients. This novel perspective assists analysis of its convergence rate and provides a new direction for more acceleration. We prove for the first time that the federated averaging algorithm is guaranteed to converge for non-convex problems, without imposing additional assumptions. We further propose a novel accelerated federated learning algorithm and provide a convergence guarantee. Simulated federated learning experiments are conducted to train deep neural networks on benchmark datasets, and experimental results show that our proposed method converges faster than previous approaches.