# Applications of fractional calculus in learned optimization

arXiv: https://arxiv.org/abs/2411.14855 · HTML: https://mlsanity.com/paper/2411.14855/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Builds on (most cited first, 10 of 10)

- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [VeLO: Training Versatile Learned Optimizers by Scaling Up](https://mlsanity.com/paper/2211.09760/) (2022)
- [A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases](https://mlsanity.com/paper/2209.11208/) (2022)
- [Transformer-Based Learned Optimization](https://mlsanity.com/paper/2212.01055/) (2022)
- [Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies](https://mlsanity.com/paper/2304.12180/) (2023)
- [The Gradient descent method from the perspective of fractional calculus](https://mlsanity.com/paper/1911.08058/) (2019)
- [A Caputo fractional derivative-based algorithm for optimization](https://mlsanity.com/paper/2104.02259/) (2021)
- [The Novel Adaptive Fractional Order Gradient Decent Algorithms Design via Robust Control](https://mlsanity.com/paper/2303.04328/) (2023)

## Abstract (arXiv)

Fractional gradient descent has been studied extensively, with a focus on its ability to extend traditional gradient descent methods by incorporating fractional-order derivatives. This approach allows for more flexibility in navigating complex optimization landscapes and offers advantages in certain types of problems, particularly those involving non-linearities and chaotic dynamics. Yet, the challenge of fine-tuning the fractional order parameters remains unsolved. In this work, we demonstrate that it is possible to train a neural network to predict the order of the gradient effectively.