Research on Optimal Control Problem Based on Reinforcement Learning under Knightian Uncertainty
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
Considering that the decision-making environment faced by reinforcement learning (RL) agents is full of Knightian uncertainty, this paper describes the exploratory state dynamics equation in Knightian uncertainty to study the entropy-regularized relaxed stochastic control problem in a Knightian uncertainty environment. By employing stochastic analysis theory and the dynamic programming principle under nonlinear expectation, we derive the Hamilton-Jacobi-Bellman (HJB) equation and solve for the optimal policy that achieves a trade-off between exploration and exploitation. Subsequently, for the linear-quadratic (LQ) case, we examine the agent's optimal randomized feedback control under both state-dependent and state-independent reward scenarios, proving that the optimal randomized feedback control follows a Gaussian distribution in the LQ framework. Furthermore, we investigate how the degree of Knightian uncertainty affects the variance of the optimal feedback policy. Additionally, we establish the solvability equivalence between non-exploratory and exploratory LQ problems under Knightian uncertainty and analyze the associated exploration cost. Finally, we provide an LQ example and validate the theoretical findings through numerical simulations.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 14 of its 48 references that are in the map. First posted on arXiv 16 Jun 2025.
Builds on
14 of 48 references.
References · 14
- 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
- 2022q-Learning in Continuous Timecited by 91
- 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 87
- 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
- 2020Entropy Regularization for Mean Field Games with Learningcited by 58
- 2021Continuous-Time Model-Based Reinforcement Learningcited by 41
- 2020Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizoncited by 35
- 2021Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex modelscited by 21
- 2020State-Dependent Temperature Control for Langevin Diffusionscited by 20
- 2024Reward-Directed Score-Based Diffusion Models via q-Learningcited by 19
- 2025Distributed Neural Policy Gradient Algorithm for Global Convergence of Networked Multi-Agent Reinforcement Learningcited by 5
- 2013Optimal stochastic control and optimal consumption and portfolio with G-Brownian motioncited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Reinforcement Learning Framework For Stochastic Optimal Control Problem Under Model Uncertainty
- 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
- 2023Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization
- 2025Optimal Control of Probabilistic Dynamics Models via Mean Hamiltonian Minimization
- 2026Discretization error from regularized Reinforcement Learning to continuous-time stochastic control
- 2025Learning-based primal-dual optimal control of discrete-time stochastic systems with multiplicative noise
- 2025A Temporal Difference Method for Stochastic Continuous Dynamics
- 2024Model-free stochastic linear quadratic design by semidefinite programming