# Path Consistency Learning in Tsallis Entropy Regularized MDPs

arXiv: https://arxiv.org/abs/1802.03501 · HTML: https://mlsanity.com/paper/1802.03501/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2018.

## Built on by (most cited first, 22 of 22)

- [Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization](https://mlsanity.com/paper/2303.15810/) (2023)
- [Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence](https://mlsanity.com/paper/2105.11066/) (2021)
- [Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learning](https://mlsanity.com/paper/2110.10719/) (2021)
- [Your Policy Regularizer is Secretly an Adversary](https://mlsanity.com/paper/2203.12592/) (2022)
- [Improved Off-policy Reinforcement Learning in Biological Sequence Design](https://mlsanity.com/paper/2410.04461/) (2024)
- [Maximum Causal Tsallis Entropy Imitation Learning](https://mlsanity.com/paper/1805.08336/) (2018)
- [Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropy](https://mlsanity.com/paper/2407.03888/) (2024)
- [Quasi-optimal Reinforcement Learning with Continuous Actions](https://mlsanity.com/paper/2301.08940/) (2023)
- [Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferences](https://mlsanity.com/paper/2602.23116/) (2026)
- [Divergence-Augmented Policy Optimization](https://mlsanity.com/paper/2501.15034/) (2025)
- [Learning accurate and interpretable tree-based models](https://mlsanity.com/paper/2405.15911/) (2024)
- [q-exponential family for policy optimization](https://mlsanity.com/paper/2408.07245/) (2024)
- [Discrete Compositional Generation via General Soft Operators and Robust Reinforcement Learning](https://mlsanity.com/paper/2506.17007/) (2025)
- [Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence](https://mlsanity.com/paper/2301.11476/) (2023)
- [On the Effect of Regularization in Policy Mirror Descent](https://mlsanity.com/paper/2507.08718/) (2025)
- [Perspectives on Tsallis Statistics for Artificial Intelligence](https://mlsanity.com/paper/2608.01223/) (2026)
- [q-Munchausen Reinforcement Learning](https://mlsanity.com/paper/2205.07467/) (2022)
- [Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learning](https://mlsanity.com/paper/2205.07885/) (2022)
- [Bridging the Gap between Newton-Raphson Method and Regularized Policy Iteration](https://mlsanity.com/paper/2310.07211/) (2023)
- [How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum](https://mlsanity.com/paper/2604.25907/) (2026)
- [Rationalizing Boltzmann Rationality: An Axiomatic Characterization of Entropy-Regularized Policies](https://mlsanity.com/paper/2607.17316/) (2026)
- [Fast Regularized Policy Mirror Descent with One-Step TD Updates](https://mlsanity.com/paper/2609.39837/) (2026)

## Abstract (arXiv)

We study the sparse entropy-regularized reinforcement learning (ERL) problem in which the entropy term is a special form of the Tsallis entropy. The optimal policy of this formulation is sparse, i.e.,~at each state, it has non-zero probability for only a small number of actions. This addresses the main drawback of the standard Shannon entropy-regularized RL (soft ERL) formulation, in which the optimal policy is softmax, and thus, may assign a non-negligible probability mass to non-optimal actions. This problem is aggravated as the number of actions is increased. In this paper, we follow the work of Nachum et al. (2017) in the soft ERL setting, and propose a class of novel path consistency learning (PCL) algorithms, called em sparse PCL, for the sparse ERL problem that can work with both on-policy and off-policy data. We first derive a em sparse consistency equation that specifies a relationship between the optimal value function and policy of the sparse ERL along any system trajectory. Crucially, a weak form of the converse is also true, and we quantify the sub-optimality of a policy which satisfies sparse consistency, and show that as we increase the number of actions, this sub-optimality is better than that of the soft ERL optimal policy. We then use this result to derive the sparse PCL algorithms. We empirically compare sparse PCL with its soft counterpart, and show its advantage, especially in problems with a large number of actions.