# Exploratory Utility Maximization Problem with Tsallis Entropy

arXiv: https://arxiv.org/abs/2502.01269 · HTML: https://mlsanity.com/paper/2502.01269/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 31 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 8 of 8)

- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Exploratory HJB equations and their convergence](https://mlsanity.com/paper/2109.10269/) (2021)
- [Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithm](https://mlsanity.com/paper/2208.02409/) (2022)
- [Data-Driven Merton's Strategies via Policy Randomization](https://mlsanity.com/paper/2312.11797/) (2023)
- [Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropy](https://mlsanity.com/paper/2407.03888/) (2024)
- [Exploratory mean-variance portfolio selection with Choquet regularizers](https://mlsanity.com/paper/2307.03026/) (2023)

## Built on by (most cited first, 1 of 1)

- [Entropy-Regularized Reinforcement Learning for Linear-Quadratic Stackelberg Differential Games in Regime-Switching Diffusion Models](https://mlsanity.com/paper/2606.28671/) (2026)

## Abstract (arXiv)

We study expected utility maximization problem with constant relative risk aversion utility function in a complete market under the reinforcement learning framework. To induce exploration, we introduce the Tsallis entropy regularizer, which generalizes the commonly used Shannon entropy. Unlike the classical Merton's problem, which is always well-posed and admits closed-form solutions, we find that the utility maximization exploratory problem is ill-posed in certain cases, due to over-exploration. With a carefully selected primary temperature function, we investigate two specific examples, for which we fully characterize their well-posedness and provide semi-closed-form solutions. It is interesting to find that one example has the well-known Gaussian distribution as the optimal strategy, while the other features the rare Wigner semicircle distribution, which is equivalent to a scaled Beta distribution. The means of the two optimal exploratory policies coincide with that of the classical counterpart. In addition, we examine the convergence of the value function and optimal exploratory strategy as the exploration vanishes. Finally, we design a reinforcement learning algorithm and conduct numerical experiments to demonstrate the advantages of reinforcement learning.