mlsanityData to 14 Oct 2026

First-order Sobolev Reinforcement Learning

arXiv 2511.19165 ↗First posted 24 Nov 2025cs.LG cs.ROcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We propose a refinement of temporal-difference learning that enforces first-order Bellman consistency: the learned value function is trained to match not only the Bellman targets in value but also their derivatives with respect to states and actions. By differentiating the Bellman backup through differentiable dynamics, we obtain analytically consistent gradient targets. Incorporating these into the critic objective using a Sobolev-type loss encourages the critic to align with both the value and local geometry of the target function. This first-order TD matching principle can be seamlessly integrated into existing algorithms, such as Q-learning or actor-critic methods (e.g., DDPG, SAC), potentially leading to faster critic convergence and more stable policy gradients without altering their overall structure.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 7 of its 11 references that are in the map. First posted on arXiv 24 Nov 2025.

Builds on

7 of 11 references.

References · 7

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  5. 2017Sobolev Training for Neural Networkscited by 143
  6. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
  7. 2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 20

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Temporal Difference Method for Stochastic Continuous Dynamics
  2. 2026Towards Parameter-Free Temporal Difference Learning
  3. 2024Multi-State TD Target for Model-Free Reinforcement Learning
  4. 2025Adversarially-Robust TD Learning with Markovian Data: Finite-Time Rates and Fundamental Limits
  5. 2026Distributional value gradients for stochastic environments
  6. 2025Towards Formalizing Reinforcement Learning Theory: A Robbins-Siegmund Approach
  7. 2026On the Convergence of Semi-Gradient TD: A Mean Field Perspective
  8. 2025Stabilizing Temporal Difference Learning via Implicit Stochastic Recursion