q-exponential family for policy optimization
Not on the map. Why?
Abstract (arXiv)
Policy optimization methods benefit from a simple and tractable policy parametrization, usually the Gaussian for continuous action spaces. In this paper, we consider a broader policy family that remains tractable: the q-exponential family. This family of policies is flexible, allowing the specification of both heavy-tailed policies (q>1) and light-tailed policies (q<1). This paper examines the interplay between q-exponential policies for several actor-critic algorithms conducted on both online and offline problems. We find that heavy-tailed policies are more effective in general and can consistently improve on Gaussian. In particular, we find the Student's t-distribution to be more stable than the Gaussian across settings and that a heavy-tailed q-Gaussian for Tsallis Advantage Weighted Actor-Critic consistently performs well in offline benchmark problems. Our code is available at urlhttps://github.com/lingweizhu/qexp.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 26 of its 55 references that are in the map. First posted on arXiv 14 Aug 2024.
Builds on
26 of 55 references.
References · 26
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
- 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
- 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
- 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
- 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
- 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
- 2018Understanding the impact of entropy on policy optimizationcited by 171
- 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
- 2019Sparse Sequence-to-Sequence Modelscited by 167
14 more
- 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
- 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
- 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
- 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
- 2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 36
- 2020Leverage the Average: an Analysis of KL Regularization in RLcited by 33
- 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
- 2018Path Consistency Learning in Tsallis Entropy Regularized MDPscited by 22
- 2019Entropic Regularization of Markov Decision Processescited by 15
- 2017Two-temperature logistic regression based on the Tsallis divergencecited by 14
- 2021Sparse Continuous Distributions and Fenchel-Young Lossescited by 13
- 2018Greedy Actor-Critic: A New Conditional Cross-Entropy Method for Policy Improvementcited by 12
- 2021On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Controlcited by 12
- 2023Quasi-optimal Reinforcement Learning with Continuous Actionscited by 7
Built on by
Citing papers per year: 2025: 2.
2 papers.