mlsanityData to 14 Oct 2026

Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settings

arXiv 2508.12327 ↗First posted 17 Aug 2025cs.LG math.OCcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we analyze the convergence properties of the Lion optimizer. First, we establish that the Lion optimizer attains a convergence rate of O(d1/2T-1/4) under standard assumptions, where d denotes the problem dimension and T is the iteration number. To further improve this rate, we introduce the Lion optimizer with variance reduction, resulting in an enhanced convergence rate of O(d1/2T-1/3). We then analyze in distributed settings, where the standard and variance reduced version of the distributed Lion can obtain the convergence rates of O(d1/2(nT)-1/4) and O(d1/2(nT)-1/3), with n denoting the number of nodes. Furthermore, we investigate a communication-efficient variant of the distributed Lion that ensures sign compression in both communication directions. By employing the unbiased sign operations, the proposed Lion variant and its variance reduction counterpart, achieve convergence rates of Oleft( max leftfracd1/4T1/4, fracd1/10n1/5T1/5 right right) and Oleft( fracd1/4T1/4 right), respectively.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 17 Aug 2025.

Builds on

18 of 26 references.

References · 18

  1. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  2. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  3. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  4. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
  5. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  6. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 287
  7. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 220
  8. 2020An Improved Analysis of Stochastic Gradient Descent with Momentumcited by 119
  9. 2020PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimizationcited by 119
  10. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  11. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
  12. 2024Communication Efficient Distributed Training with Distributed Lioncited by 18
6 more
  1. 2024Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reductioncited by 13
  2. 2024Convergence Rate Analysis of LIONcited by 11
  3. 2021Escaping Saddle Points Faster with Stochastic Momentumcited by 8
  4. 2024FedLion: Faster Adaptive Federated Optimization with Fewer Communicationcited by 6
  5. 2025Better Convergence Guarantees for Sign-Based Momentum Methodscited by 6
  6. 2024Lion Cub: Minimizing Communication Overhead in Distributed Lioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
  2. 2026OLion: Approaching the Hadamard Ideal by Intersecting Spectral and ℓ∞ Implicit Biasescited by 3
  3. 2026When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Boundscited by 2
  4. 2026CLion: Efficient Cautious Lion Optimizer with Enhanced Generalizationcited by 0
  5. 2026Accelerating LMO-Based Optimization via Implicit Gradient Transportcited by 0
  6. 2026ProofLoom: Proof-Obligation-Driven Theory Construction for Autoformalizing Research-Level Stochastic Optimizationcited by 0