# Fractional Deep Neural Network via Constrained Optimization

arXiv: https://arxiv.org/abs/2004.00719 · HTML: https://mlsanity.com/paper/2004.00719/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2020.

## Built on by (most cited first, 13 of 13)

- [Unleashing the Potential of Fractional Calculus in Graph Neural Networks with FROND](https://mlsanity.com/paper/2404.17099/) (2024)
- [Novel Deep neural networks for solving Bayesian statistical inverse](https://mlsanity.com/paper/2102.03974/) (2021)
- [Coupling Graph Neural Networks with Fractional Order Continuous Dynamics: A Robustness Study](https://mlsanity.com/paper/2401.04331/) (2024)
- [Differential Equations for Continuous-Time Deep Learning](https://mlsanity.com/paper/2401.03965/) (2024)
- [Novel DNNs for Stiff ODEs with Applications to Chemically Reacting Flows](https://mlsanity.com/paper/2104.01914/) (2021)
- [Neural Variable-Order Fractional Differential Equation Networks](https://mlsanity.com/paper/2503.16207/) (2025)
- [An Optimal Time Variable Learning Framework for Deep Neural Networks](https://mlsanity.com/paper/2204.08528/) (2022)
- [Efficient Training of Neural Fractional-Order Differential Equation via Adjoint Backpropagation](https://mlsanity.com/paper/2503.16666/) (2025)
- [NINNs: Nudging Induced Neural Networks](https://mlsanity.com/paper/2203.07947/) (2022)
- [Lecture Notes: Neural Network Architectures](https://mlsanity.com/paper/2304.05133/) (2023)
- [Neural Network Representation of Time Integrators](https://mlsanity.com/paper/2211.17039/) (2022)
- [Time Regularization in Optimal Time Variable Learning](https://mlsanity.com/paper/2306.16111/) (2023)
- [OCTANE -- Optimal Control for Tensor-based Autoencoder Network Emergence: Explicit Case](https://mlsanity.com/paper/2509.08169/) (2025)

## Abstract (arXiv)

This paper introduces a novel algorithmic framework for a deep neural network (DNN), which in a mathematically rigorous manner, allows us to incorporate history (or memory) into the network -- it ensures all layers are connected to one another. This DNN, called Fractional-DNN, can be viewed as a time-discretization of a fractional in time nonlinear ordinary differential equation (ODE). The learning problem then is a minimization problem subject to that fractional ODE as constraints. We emphasize that an analogy between the existing DNN and ODEs, with standard time derivative, is well-known by now. The focus of our work is the Fractional-DNN. Using the Lagrangian approach, we provide a derivation of the backward propagation and the design equations. We test our network on several datasets for classification problems. Fractional-DNN offers various advantages over the existing DNN. The key benefits are a significant improvement to the vanishing gradient issue due to the memory effect, and better handling of nonsmooth data due to the network's ability to approximate non-smooth functions.