# Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropy

arXiv: https://arxiv.org/abs/2407.03888 · HTML: https://mlsanity.com/paper/2407.03888/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 12 of its 48 references that are in the map.
- First posted on arXiv 4 Jul 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 12 of 12)

- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Making Deep Q-learning methods robust to time discretization](https://mlsanity.com/paper/1901.09732/) (2019)
- [Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learning](https://mlsanity.com/paper/1709.06293/) (2017)
- [Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems](https://mlsanity.com/paper/2211.00617/) (2022)
- [Path Consistency Learning in Tsallis Entropy Regularized MDPs](https://mlsanity.com/paper/1802.03501/) (2018)
- [Continuous-time q-learning for mean-field control problems](https://mlsanity.com/paper/2306.16208/) (2023)
- [Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learning](https://mlsanity.com/paper/1902.00137/) (2019)
- [On optimal tracking portfolio in incomplete markets: The reinforcement learning approach](https://mlsanity.com/paper/2311.14318/) (2023)
- [MALA-within-Gibbs samplers for high-dimensional distributions with sparse conditional structure](https://mlsanity.com/paper/1908.09429/) (2019)
- [Reinforcement Learning for Intensity Control: An Application to Choice-Based Network Revenue Management](https://mlsanity.com/paper/2406.05358/) (2024)

## Built on by (most cited first, 8 of 8)

- [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024)
- [Continuous-time reinforcement learning for optimal switching over multiple regimes](https://mlsanity.com/paper/2512.04697/) (2025)
- [Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems](https://mlsanity.com/paper/2606.11798/) (2026)
- [Robust Reinforcement Learning under Diffusion Models for Data with Jumps](https://mlsanity.com/paper/2411.11697/) (2024)
- [Exploratory Utility Maximization Problem with Tsallis Entropy](https://mlsanity.com/paper/2502.01269/) (2025)
- [Continuous-time q-learning for Markov regime switching system under Tsallis entropy](https://mlsanity.com/paper/2601.19299/) (2026)
- [Entropy-Regularized Reinforcement Learning for Linear-Quadratic Stackelberg Differential Games in Regime-Switching Diffusion Models](https://mlsanity.com/paper/2606.28671/) (2026)
- [An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flows](https://mlsanity.com/paper/2604.05398/) (2026)

## Abstract (arXiv)

This paper studies the continuous-time reinforcement learning in jump-diffusion models by featuring the q-learning (the continuous-time counterpart of Q-learning) under Tsallis entropy regularization. Contrary to the Shannon entropy, the general form of Tsallis entropy renders the optimal policy not necessarily a Gibbs measure. Herein, the Lagrange multiplier and KKT condition are needed to ensure that the learned policy is a probability density function. As a consequence, the characterization of the optimal policy using the q-function also involves a Lagrange multiplier. In response, we establish the martingale characterization of the q-function and devise two q-learning algorithms depending on whether the Lagrange multiplier can be derived explicitly or not. In the latter case, we consider different parameterizations of the optimal q-function and the optimal policy, and update them alternatively in an Actor-Critic manner. We also study two numerical examples, namely, an optimal liquidation problem in dark pools and a non-LQ control problem. It is interesting to see therein that the optimal policies under the Tsallis entropy regularization can be characterized explicitly, which are distributions concentrated on some compact support. The satisfactory performance of our q-learning algorithms is illustrated in each example.