# Continuous-time reinforcement learning for optimal switching over multiple regimes

arXiv: https://arxiv.org/abs/2512.04697 · HTML: https://mlsanity.com/paper/2512.04697/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 4 Dec 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 19 of 19)

- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Exploratory HJB equations and their convergence](https://mlsanity.com/paper/2109.10269/) (2021)
- [Continuous-time q-learning for mean-field control problems](https://mlsanity.com/paper/2306.16208/) (2023)
- [Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithm](https://mlsanity.com/paper/2208.02409/) (2022)
- [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024)
- [Policy Iteration for Exploratory Hamilton--Jacobi--Bellman Equations](https://mlsanity.com/paper/2406.00612/) (2024)
- [Regret of exploratory policy improvement and q-learning](https://mlsanity.com/paper/2411.01302/) (2024)
- [Convergence of Policy Iteration for Entropy-Regularized Stochastic Control Problems](https://mlsanity.com/paper/2209.07059/) (2022)
- [On optimal tracking portfolio in incomplete markets: The reinforcement learning approach](https://mlsanity.com/paper/2311.14318/) (2023)
- [Learning to Optimally Stop Diffusion Processes, with Financial Applications](https://mlsanity.com/paper/2408.09242/) (2024)
- [Exploratory Optimal Stopping: A Singular Control Formulation](https://mlsanity.com/paper/2408.09335/) (2024)
- [Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching](https://mlsanity.com/paper/2404.17939/) (2024)
- [Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropy](https://mlsanity.com/paper/2407.03888/) (2024)
- [A Two-fold Randomization Framework for Impulse Control Problems](https://mlsanity.com/paper/2509.12018/) (2025)
- [A Reinforcement Learning Framework for Some Singular Stochastic Control Problems](https://mlsanity.com/paper/2506.22203/) (2025)
- [Management strategies for run-of-river hydropower plants -an optimal switching approach](https://mlsanity.com/paper/2009.10554/) (2020)
- [Reinforcement learning for irreversible reinsurance problems: the randomized singular control approach](https://mlsanity.com/paper/2512.02769/) (2025)

## Built on by (most cited first, 6 of 6)

- [Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations](https://mlsanity.com/paper/2604.27372/) (2026)
- [Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms](https://mlsanity.com/paper/2604.27378/) (2026)
- [Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems](https://mlsanity.com/paper/2606.11798/) (2026)
- [Randomized Optimal Switching Problem and Related Mirror Descent Flow](https://mlsanity.com/paper/2606.12875/) (2026)
- [Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning](https://mlsanity.com/paper/2607.03168/) (2026)
- [Continuous-Time Reinforcement Learning for N-Player Stochastic Differential Games with Exploratory Policies](https://mlsanity.com/paper/2607.19928/) (2026)

## Abstract (arXiv)

This paper studies the continuous-time reinforcement learning (RL) for optimal switching problems across multiple regimes. We consider a type of exploratory formulation under entropy regularization where the agent randomizes both the timing of switches and the selection of regimes through the generator matrix of an associated continuous-time finite-state Markov chain. We establish the well-posedness of the associated system of Hamilton-Jacobi-Bellman (HJB) equations and provide a characterization of the optimal policy. The policy improvement and the convergence of the policy iterations are rigorously established by analyzing the system of equations. We also show that the value function in the exploratory formulation converges to the one in the classical formulation as the temperature parameter vanishes. Finally, a model-free reinforcement learning algorithm is devised and implemented by invoking the policy evaluation based on the martingale characterization. Our numerical examples with financial applications illustrate the effectiveness and efficiency of the proposed RL algorithm.