mlsanityData to 14 Oct 2026

Is Bellman Equation Enough for Learning Control?

arXiv 2503.02171 ↗First posted 4 Mar 2025cs.LG math.OCcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

The Bellman equation and its continuous-time counterpart, the Hamilton-Jacobi-Bellman (HJB) equation, serve as necessary conditions for optimality in reinforcement learning and optimal control. While the value function is known to be the unique solution to the Bellman equation in tabular settings, we demonstrate that this uniqueness fails to hold in continuous state spaces. Specifically, for linear dynamical systems, we prove the Bellman equation admits at least binom2nn solutions, where n is the state dimension. Crucially, only one of these solutions yields both an optimal policy and a stable closed-loop system. We then demonstrate a common failure mode in value-based methods: convergence to unstable solutions due to the exponential imbalance between admissible and inadmissible solutions. Finally, we introduce a positive-definite neural architecture that guarantees convergence to the stable solution by construction to address this issue.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 40 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

19 of 40 references.

References · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2017Self-Normalizing Neural Networkscited by 598
  5. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  6. 2019Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learningcited by 212
  7. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
  8. 2022Safe Control with Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction methodscited by 151
  9. 2020Learning Stable Deep Dynamics Modelscited by 111
  10. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
  11. 2019Adaptive Deep Learning for High-Dimensional Hamilton-Jacobi-Bellman Equationscited by 40
  12. 2021Stabilizing Neural Control Using Self-Learned Almost Lyapunov Criticscited by 35
7 more
  1. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  2. 2021Value Iteration in Continuous Actions, States and Timecited by 18
  3. 2019HJB Optimal Feedback Control with Deep Differential Value Functions and Action Constraintscited by 8
  4. 2021Robust Value Iteration for Continuous Control Taskscited by 8
  5. 2021Continuous-Time Fitted Value Iteration for Robust Policiescited by 6
  6. 2024On the Uniqueness of Solution for the Bellman Equation of LTL Objectivescited by 3
  7. 2015On nouniqueness of solutions of Hamilton-Jacobi-Bellman equationscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Temporal Difference Method for Stochastic Continuous Dynamics
  2. 2024On the stability of Lipschitz continuous control problems and its application to reinforcement learning
  3. 2025Closed Form HJB Solution for Continuous-Time Optimal Control of a Non-Linear Input-Affine System
  4. 2025Continuous-Time Value Iteration for Multi-Agent Reinforcement Learning
  5. 2025Theoretical Barriers in Bellman-Based Reinforcement Learning
  6. 2025Bridging Continuous-time LQR and Reinforcement Learning via Gradient Flow of the Bellman Error
  7. 2025Local-Global Learning of Interpretable Control Policies: The Interface between MPC and Reinforcement Learning
  8. 2025Bellman operator convergence enhancements in reinforcement learning algorithms