# Continuous-time q-learning for Markov regime switching system under Tsallis entropy

arXiv: https://arxiv.org/abs/2601.19299 · HTML: https://mlsanity.com/paper/2601.19299/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 12 of 12)

- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Making Deep Q-learning methods robust to time discretization](https://mlsanity.com/paper/1901.09732/) (2019)
- [Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learning](https://mlsanity.com/paper/1709.06293/) (2017)
- [Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls](https://mlsanity.com/paper/2010.14087/) (2020)
- [Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems](https://mlsanity.com/paper/2211.00617/) (2022)
- [Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learning](https://mlsanity.com/paper/1902.00137/) (2019)
- [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024)
- [Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropy](https://mlsanity.com/paper/2407.03888/) (2024)
- [Exploratory Mean-Variance Portfolio Optimization with Regime-Switching Market Dynamics](https://mlsanity.com/paper/2501.16659/) (2025)

## Built on by (most cited first, 1 of 1)

- [Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning](https://mlsanity.com/paper/2607.03168/) (2026)

## Abstract (arXiv)

This paper studies continuous-time q-learning (the continuous-time counterpart of Q-learning) for a Markov regime-switching system under Tsallis entropy regularization. The Tsallis entropy regularization yields an optimal policy distribution that may not necessarily be a Gibbs measure, thereby complicating algorithm design. Furthermore, to address the limited universality of current continuous-time regime-switching reinforcement learning algorithms (often restricted to the exploratory mean-variance framework), this study focuses on continuous-time q-learning for Markov regime-switching systems based on Tsallis entropy, aiming for a more universally applicable continuous-time reinforcement learning method. We establish the martingale characterization of the q-function under Tsallis entropy for continuous-time Markov regime-switching systems. We further design two q-learning algorithms that differ based on whether the Lagrange multiplier can be explicitly derived. We apply these algorithms to the continuous-time exploratory mean-variance portfolio optimization problem in a regime-switching market. Numerical experiments demonstrate the satisfactory performance of our q-learning algorithms.