# Fractional Optimizers Meet Fractal Activation Functions: An Empirical Study of Multi-Scale Optimization in Neural Network

arXiv: https://arxiv.org/abs/2608.14636 · HTML: https://mlsanity.com/paper/2608.14636/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 82 references that are in the map.
- First posted on arXiv 28 Jul 2026.

## Builds on (most cited first, 3 of 3)

- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [The boundary of neural network trainability is fractal](https://mlsanity.com/paper/2402.06184/) (2024)
- [Effective Dimension Aware Fractional-Order Stochastic Gradient Descent for Convex Optimization Problems](https://mlsanity.com/paper/2503.13764/) (2025)

## Abstract (arXiv)

Fractional optimization methods and fractal activation functions are two independent directions for improving neural network training. Fractional optimizers extend first-order optimization through fractional derivatives and memory effects, whereas fractal activations introduce multi-scale nonlinear representations based on self-similar Weierstrass- and Blancmange-type functions. Here, we investigate their interaction within a unified experimental framework. We evaluate fractional optimizer families on Ackley and Himmelblau benchmark surfaces, in standard form and with additive Weierstrass-type perturbations, and then in feed-forward neural networks with conventional and fractal activations on ten classification datasets. The comparison includes standard methods, regularization-style optimizers, explicit and adaptive memory-based fractional optimizers, and other representative literature methods. Overall, fractional optimization and fractal activations show useful but selective pairings. Regularization-style fractional scaling performs well with selected fractal activations in network training, while Gr"unwald--Letnikov memory is most relevant on perturbed surfaces. Adaptive memory improves plain memory substitution in several cases, supporting controlled fractional memory as a promising direction rather than a universal replacement.