# Research on Optimal Control Problem Based on Reinforcement Learning under Knightian Uncertainty

arXiv: https://arxiv.org/abs/2506.13207 · HTML: https://mlsanity.com/paper/2506.13207/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 48 references that are in the map.
- First posted on arXiv 16 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 14 of 14)

- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity](https://mlsanity.com/paper/2208.05767/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Entropy Regularization for Mean Field Games with Learning](https://mlsanity.com/paper/2010.00145/) (2020)
- [Continuous-Time Model-Based Reinforcement Learning](https://mlsanity.com/paper/2102.04764/) (2021)
- [Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon](https://mlsanity.com/paper/2006.15316/) (2020)
- [Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models](https://mlsanity.com/paper/2112.10264/) (2021)
- [State-Dependent Temperature Control for Langevin Diffusions](https://mlsanity.com/paper/2011.07456/) (2020)
- [Reward-Directed Score-Based Diffusion Models via q-Learning](https://mlsanity.com/paper/2409.04832/) (2024)
- [Distributed Neural Policy Gradient Algorithm for Global Convergence of Networked Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2505.24113/) (2025)
- [Optimal stochastic control and optimal consumption and portfolio with G-Brownian motion](https://mlsanity.com/paper/1309.0209/) (2013)
- [A Central Limit Theorem, Loss Aversion and Multi-Armed Bandits](https://mlsanity.com/paper/2106.05472/) (2021)
- [Ergodic Risk Sensitive Control of Markovian Multiclass Many-Server Queues with Abandonment](https://mlsanity.com/paper/2410.17205/) (2024)

## Abstract (arXiv)

Considering that the decision-making environment faced by reinforcement learning (RL) agents is full of Knightian uncertainty, this paper describes the exploratory state dynamics equation in Knightian uncertainty to study the entropy-regularized relaxed stochastic control problem in a Knightian uncertainty environment. By employing stochastic analysis theory and the dynamic programming principle under nonlinear expectation, we derive the Hamilton-Jacobi-Bellman (HJB) equation and solve for the optimal policy that achieves a trade-off between exploration and exploitation. Subsequently, for the linear-quadratic (LQ) case, we examine the agent's optimal randomized feedback control under both state-dependent and state-independent reward scenarios, proving that the optimal randomized feedback control follows a Gaussian distribution in the LQ framework. Furthermore, we investigate how the degree of Knightian uncertainty affects the variance of the optimal feedback policy. Additionally, we establish the solvability equivalence between non-exploratory and exploratory LQ problems under Knightian uncertainty and analyze the associated exploration cost. Finally, we provide an LQ example and validate the theoretical findings through numerical simulations.