# Non-Markovian Reinforcement Learning using Fractional Dynamics

arXiv: https://arxiv.org/abs/2107.13790 · HTML: https://mlsanity.com/paper/2107.13790/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 44 references that are in the map.
- First posted on arXiv 29 Jul 2021.

## Builds on (most cited first, 10 of 10)

- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Reinforcement Learning with Non-Markovian Rewards](https://mlsanity.com/paper/1912.02552/) (2019)
- [Blending MPC & Value Function Approximation for Efficient Reinforcement Learning](https://mlsanity.com/paper/2012.05909/) (2020)
- [Reinforcement Learning for Joint Optimization of Multiple Rewards](https://mlsanity.com/paper/1909.02940/) (2019)
- [Nonparametric General Reinforcement Learning](https://mlsanity.com/paper/1611.08944/) (2016)
- [Dealing with Unknown Unknowns: Identification and Selection of Minimal Sensing for Fractional Dynamics with Unknown Inputs](https://mlsanity.com/paper/1803.04866/) (2018)
- [Re-thinking EEG-based non-invasive brain interfaces: modeling and analysis](https://mlsanity.com/paper/1803.10318/) (2018)
- [Learning Latent Fractional dynamics with Unknown Unknowns](https://mlsanity.com/paper/1811.00703/) (2018)
- [Non-Markovian Control with Gated End-to-End Memory Policy Networks](https://mlsanity.com/paper/1705.10993/) (2017)

## Built on by (most cited first, 5 of 5)

- [Fractional dynamics foster deep learning of COPD stage prediction](https://mlsanity.com/paper/2303.07537/) (2023)
- [Constructing Non-Markovian Decision Process via History Aggregator](https://mlsanity.com/paper/2506.24026/) (2025)
- [Model-Based Reinforcement Learning in Discrete-Action Non-Markovian Reward Decision Processes](https://mlsanity.com/paper/2512.14617/) (2025)
- [A Data-Integrated Framework for Learning Fractional-Order Nonlinear Dynamical Systems](https://mlsanity.com/paper/2506.15665/) (2025)
- [Lattice Annotated Temporal (LAT) Logic for Non-Markovian Reasoning](https://mlsanity.com/paper/2509.02958/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) is a technique to learn the control policy for an agent that interacts with a stochastic environment. In any given state, the agent takes some action, and the environment determines the probability distribution over the next state as well as gives the agent some reward. Most RL algorithms typically assume that the environment satisfies Markov assumptions (i.e. the probability distribution over the next state depends only on the current state). In this paper, we propose a model-based RL technique for a system that has non-Markovian dynamics. Such environments are common in many real-world applications such as in human physiology, biological systems, material science, and population dynamics. Model-based RL (MBRL) techniques typically try to simultaneously learn a model of the environment from the data, as well as try to identify an optimal policy for the learned model. We propose a technique where the non-Markovianity of the system is modeled through a fractional dynamical system. We show that we can quantify the difference in the performance of an MBRL algorithm that uses bounded horizon model predictive control from the optimal policy. Finally, we demonstrate our proposed framework on a pharmacokinetic model of human blood glucose dynamics and show that our fractional models can capture distant correlations on real-world datasets.