# Boundary-aware Reinforcement Learning for Hypercube State Spaces via Deterministic Policy Gradient

arXiv: https://arxiv.org/abs/2610.09712 · HTML: https://mlsanity.com/paper/2610.09712/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 6 of its 41 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 6 of 6)

- [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024)
- [Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State](https://mlsanity.com/paper/2509.23711/) (2025)
- [Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations](https://mlsanity.com/paper/2604.27372/) (2026)
- [Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms](https://mlsanity.com/paper/2604.27378/) (2026)
- [Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems](https://mlsanity.com/paper/2606.11798/) (2026)
- [Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies](https://mlsanity.com/paper/2607.11005/) (2026)

## Abstract (arXiv)

We develop a continuous-time deterministic policy gradient framework for reinforcement learning with reflected state dynamics, where the state process is governed by a controlled reflected stochastic differential equation on a hypercube. Under suitable regularity assumptions, we establish the connection between the value function and the Neumann Bellman equation, introduce an advantage-rate function that yields a deterministic policy gradient formula, and prove the martingale characterization theorem. Motivated by these theoretical results, we propose a continuous-time deep deterministic policy gradient algorithm for reflected stochastic systems, in which the Neumann boundary condition is imposed via either soft penalization or hard architectural constraint. We further quantify the discrepancy between the ideal continuous-time dynamics and the discretely sampled exploratory dynamics executed in practice, showing that the error decays as the time grid is refined and exploration noise vanishes. Our experiments on reservoir control problems illustrate the effectiveness of the RL framework, highlighting that boundary-aware methods substantially reduce Neumann boundary residuals and enhance learning stability.