Continuous-time q-learning for Markov regime switching system under Tsallis entropy
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
This paper studies continuous-time q-learning (the continuous-time counterpart of Q-learning) for a Markov regime-switching system under Tsallis entropy regularization. The Tsallis entropy regularization yields an optimal policy distribution that may not necessarily be a Gibbs measure, thereby complicating algorithm design. Furthermore, to address the limited universality of current continuous-time regime-switching reinforcement learning algorithms (often restricted to the exploratory mean-variance framework), this study focuses on continuous-time q-learning for Markov regime-switching systems based on Tsallis entropy, aiming for a more universally applicable continuous-time reinforcement learning method. We establish the martingale characterization of the q-function under Tsallis entropy for continuous-time Markov regime-switching systems. We further design two q-learning algorithms that differ based on whether the Lagrange multiplier can be explicitly derived. We apply these algorithms to the continuous-time exploratory mean-variance portfolio optimization problem in a regime-switching market. Numerical experiments demonstrate the satisfactory performance of our q-learning algorithms.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 29 references that are in the map. First posted on arXiv 27 Jan 2026.
Builds on
12 of 29 references.
Closest ancestors · 1
Same direction, newest first
Other references · 11
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
- 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
- 2022q-Learning in Continuous Timecited by 91
- 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
- 2019Making Deep Q-learning methods robust to time discretizationcited by 58
- 2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 37
- 2020Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controlscited by 31
- 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 25
- 2019Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learningcited by 19
- 2024Unified continuous-time q-learning for mean-field game and mean-field control problemscited by 13
- 2025Exploratory Mean-Variance Portfolio Optimization with Regime-Switching Market Dynamicscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Continuous-time reinforcement learning for optimal switching over multiple regimes
- 2025Data-Driven Long-Term Asset Allocation with Tsallis Entropy Regularization ↗
- 2026Reinforcement Learning for Continuous-Time Jump Markov Decision Processes with Applications to Network Dynamic Pricing
- 2025Exploratory Utility Maximization Problem with Tsallis Entropy
- 2024Reinforcement Learning for Jump-Diffusions, with Financial Applications
- 2026Reinforcement Learning for Dividend Optimization in Partially Observed Regime-Switching Diffusion Model
- 2024Finite-Time Error Analysis of Soft Q-Learning: Switching System Approach
- 2025Reinforcement Learning in Switching Non-Stationary Markov Decision Processes: Algorithms and Convergence Analysis