# Temporal Regularization in Markov Decision Process

arXiv: https://arxiv.org/abs/1811.00429 · HTML: https://mlsanity.com/paper/1811.00429/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2018.

## Built on by (most cited first, 3 of 3)

- [L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learning](https://mlsanity.com/paper/2202.07152/) (2022)
- [Robust Imitation of a Few Demonstrations with a Backwards Model](https://mlsanity.com/paper/2210.09337/) (2022)
- [HG2P: Hippocampus-inspired High-reward Graph and Model-Free Q-Gradient Penalty for Path Planning and Motion Control](https://mlsanity.com/paper/2410.09505/) (2024)

## Abstract (arXiv)

Several applications of Reinforcement Learning suffer from instability due to high variance. This is especially prevalent in high dimensional domains. Regularization is a commonly used technique in machine learning to reduce variance, at the cost of introducing some bias. Most existing regularization techniques focus on spatial (perceptual) regularization. Yet in reinforcement learning, due to the nature of the Bellman equation, there is an opportunity to also exploit temporal regularization based on smoothness in value estimates over trajectories. This paper explores a class of methods for temporal regularization. We formally characterize the bias induced by this technique using Markov chain concepts. We illustrate the various characteristics of temporal regularization via a sequence of simple discrete and continuous MDPs, and show that the technique provides improvement even in high-dimensional Atari games.