mlsanityData to 14 Oct 2026

Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems

arXiv 2606.11798 ↗First posted 10 Jun 2026q-fin.CP cs.LG math.OCcited by 3

On the map: Deep reinforcement learning methods → Adversarial robustness and RL in finance

Abstract (arXiv)

In this paper, we develop a continuous-time model-free reinforcement learning algorithm to learn deterministic equilibrium policies in general time-inconsistent control problems. Utilizing the extended Hamilton-Jacobi-Bellman system, we recast the original time-inconsistent problem into an equivalent two-stage problem. In the first stage, for given auxiliary functions, we employ the deterministic policy gradient approach to learn an optimal policy in an auxiliary time-consistent control problem. In the second stage, given the updated policy, we exploit the inner fixed point iterations and some martingale characterizations to learn the auxiliary functions. As a theoretical contribution, we provide some mild model assumptions and establish the convergence of inner fixed point iterations. By repeating this actor-critic style of iterations across two stages, our algorithm aims to learn the equilibrium under different sources of time-inconsistency in a unified manner. The superior effectiveness of the proposed algorithm are illustrated in two classical financial applications with time-inconsistency: mean-variance portfolio management and optimal tracking portfolio under non-exponential discounting.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 33 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

18 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Learning to Optimally Stop Diffusion Processes, with Financial Applications17 Aug

Other references · 17

  1. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  2. 2022q-Learning in Continuous Timecited by 91
  3. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
  4. 2020Entropy Regularization for Mean Field Games with Learningcited by 58
  5. 2021Exploratory HJB equations and their convergencecited by 32
  6. 2023Continuous-time q-learning for mean-field control problemscited by 22
  7. 2022Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithmcited by 17
  8. 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learningcited by 13
  9. 2024Entropy annealing for policy mirror descent in continuous time and spacecited by 12
  10. 2023On optimal tracking portfolio in incomplete markets: The reinforcement learning approachcited by 9
  11. 2024Exploratory Optimal Stopping: A Singular Control Formulationcited by 9
  12. 2024Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropycited by 8
5 more
  1. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and Statecited by 8
  2. 2025Continuous-time reinforcement learning for optimal switching over multiple regimescited by 6
  3. 2025A Two-fold Randomization Framework for Impulse Control Problemscited by 3
  4. 2026Policy Iteration Achieves Regularized Equilibrium under Time Inconsistencycited by 2
  5. 2026Equilibrium under Time-Inconsistency: A New Existence Theory by Vanishing Entropy Regularizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Actor-Critic Learning for Extended Mean Field Control with Deterministic Policiescited by 1
  2. 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergencecited by 0
  3. 2026Boundary-aware Reinforcement Learning for Hypercube State Spaces via Deterministic Policy Gradientcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploratory Mean-Variance with Jumps: An Equilibrium Approach
  2. 2026An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flows
  3. 2024Reinforcement Learning for Jump-Diffusions, with Financial Applications
  4. 2026Reinforcement Learning for Risk-Sensitive Investment Management: a Free Energy--Entropy Duality Approach
  5. 2026Reinforcement Learning for Dividend Optimization in Partially Observed Regime-Switching Diffusion Model
  6. 2024Pontryagin-Guided Policy Optimization for Merton's Portfolio Problem ↗
  7. 2026Dynamic mean-variance portfolio selection with no-shorting constraints and unknown investment opportunity sets
  8. 2024Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching