mlsanityData to 7 Oct 2026

Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learning

arXiv 1709.06293 ↗First posted 19 Sep 2017cs.LG cs.AI stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

In this paper, a sparse Markov decision process (MDP) with novel causal sparse Tsallis entropy regularization is proposed.The proposed policy regularization induces a sparse and multi-modal optimal policy distribution of a sparse MDP. The full mathematical analysis of the proposed sparse MDP is provided.We first analyze the optimality condition of a sparse MDP. Then, we propose a sparse value iteration method which solves a sparse MDP and then prove the convergence and optimality of sparse value iteration using the Banach fixed point theorem. The proposed sparse MDP is compared to soft MDPs which utilize causal entropy regularization. We show that the performance error of a sparse MDP has a constant bound, while the error of a soft MDP increases logarithmically with respect to the number of actions, where this performance error is caused by the introduced regularization term. In experiments, we apply sparse MDPs to reinforcement learning problems. The proposed method outperforms existing methods in terms of the convergence speed and performance.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Sep 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 7; 2022: 6; 2023: 7; 2024: 8; 2025: 0; 2026: 6.

36 papers.

Sort:

By area:

  1. 2019Imitation Learning as f-Divergence Minimizationcited by 121
  2. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
  3. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
  4. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 46
  5. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 29
  6. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  7. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  8. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learningcited by 16
  9. 2022KL-Entropy-Regularized RL with a Generative Model is Minimax Optimalcited by 13
  10. 2018Maximum Causal Tsallis Entropy Imitation Learningcited by 10
  11. 2023Quasi-optimal Reinforcement Learning with Continuous Actionscited by 7
  12. 2024Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropycited by 7
24 more
  1. 2022A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Searchcited by 6
  2. 2024Hard Prompts Made Interpretable: Sparse Entropy Regularization for Prompt Tuning with RLcited by 6
  3. 2024Roping in Uncertainty: Robustness and Regularization in Markov Gamescited by 5
  4. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferencescited by 5
  5. 2021Robust Entropy-regularized Markov Decision Processescited by 4
  6. 2021Planning and Learning Using Adaptive Entropy Tree Searchcited by 3
  7. 2021Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularizationcited by 2
  8. 2024q-exponential family for policy optimizationcited by 2
  9. 2024Learning Personalized Prompts for Healthcare Guidancecited by 2
  10. 2023Generalized Munchausen Reinforcement Learning using Tsallis KL Divergencecited by 1
  11. 2023Differentiable Arbitrating in Zero-sum Markov Gamescited by 1
  12. 2023Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularizationcited by 1
  13. 2024Regularized Q-Learning with Linear Function Approximationcited by 1
  14. 2024Optimal Rates of Convergence for Entropy Regularization in Discounted Markov Decision Processescited by 1
  15. 2026Perspectives on Tsallis Statistics for Artificial Intelligencecited by 1
  16. 2022q-Munchausen Reinforcement Learningcited by 0
  17. 2022Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learningcited by 0
  18. 2022Minimum Description Length Controlcited by 0
  19. 2023Bridging the Gap between Newton-Raphson Method and Regularized Policy Iterationcited by 0
  20. 2024Tsallis Entropy Regularization for Linear Quadratic Regulator and Kullback-Leibler Controlcited by 0
  21. 2026A New Kernel Regularity Condition for Distributed Mirror Descent: Broader Coverage and Simpler Analysiscited by 0
  22. 2026How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuumcited by 0
  23. 2026Rationalizing Boltzmann Rationality: An Axiomatic Characterization of Entropy-Regularized Policiescited by 0
  24. 2026Fast Regularized Policy Mirror Descent with One-Step TD Updatescited by 0