# On the continuity and smoothness of the value function in reinforcement learning and optimal control

arXiv: https://arxiv.org/abs/2403.14432 · HTML: https://mlsanity.com/paper/2403.14432/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 1 of its 15 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 1 of 1)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)

## Built on by (most cited first, 2 of 2)

- [Q-Measure-Learning for Continuous State RL: Efficient Implementation and Convergence](https://mlsanity.com/paper/2603.03523/) (2026)
- [Redesigning Regularization for Effective Policy Smoothing](https://mlsanity.com/paper/2606.13169/) (2026)

## Abstract (arXiv)

The value function plays a crucial role as a measure for the cumulative future reward an agent receives in both reinforcement learning and optimal control. It is therefore of interest to study how similar the values of neighboring states are, i.e., to investigate the continuity of the value function. We do so by providing and verifying upper bounds on the value function's modulus of continuity. Additionally, we show that the value function is always H"older continuous under relatively weak assumptions on the underlying system and that non-differentiable value functions can be made differentiable by slightly "disturbing" the system.