# Convergence Analysis of Fractional Gradient Descent

arXiv: https://arxiv.org/abs/2311.18426 · HTML: https://mlsanity.com/paper/2311.18426/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 26 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 8 of 8)

- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [On the Fenchel Duality between Strong Convexity and Lipschitz Continuous Gradient](https://mlsanity.com/paper/1803.06573/) (2018)
- [Convolutional neural networks with fractional order gradient method](https://mlsanity.com/paper/1905.05336/) (2019)
- [Lyapunov functions for nabla discrete fractional order systems](https://mlsanity.com/paper/1812.11368/) (2018)
- [Optimization and Learning With Nonlocal Calculus](https://mlsanity.com/paper/2012.07013/) (2020)
- [The Gradient descent method from the perspective of fractional calculus](https://mlsanity.com/paper/1911.08058/) (2019)
- [A Caputo fractional derivative-based algorithm for optimization](https://mlsanity.com/paper/2104.02259/) (2021)
- [On the Quality of First-Order Approximation of Functions with H"older Continuous Gradient](https://mlsanity.com/paper/2001.07946/) (2020)

## Abstract (arXiv)

Fractional derivatives are a well-studied generalization of integer order derivatives. Naturally, for optimization, it is of interest to understand the convergence properties of gradient descent using fractional derivatives. Convergence analysis of fractional gradient descent is currently limited both in the methods analyzed and the settings analyzed. This paper aims to fill in these gaps by analyzing variations of fractional gradient descent in smooth and convex, smooth and strongly convex, and smooth and non-convex settings. First, novel bounds will be established bridging fractional and integer derivatives. Then, these bounds will be applied to the aforementioned settings to prove linear convergence for smooth and strongly convex functions and O(1/T) convergence for smooth and convex functions. Additionally, we prove O(1/T) convergence for smooth and non-convex functions using an extended notion of smoothness - H"older smoothness - that is more natural for fractional derivatives. Finally, empirical results will be presented on the potential speed up of fractional gradient descent over standard gradient descent as well as some preliminary theoretical results explaining this speed up.