mlsanityData to 7 Oct 2026

Maximum Entropy Reinforcement Learning with Mixture Policies

arXiv 2103.10176 ↗First posted 18 Mar 2021cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Mixture models are an expressive hypothesis class that can approximate a rich set of policies. However, using mixture policies in the Maximum Entropy (MaxEnt) framework is not straightforward. The entropy of a mixture model is not equal to the sum of its components, nor does it have a closed-form expression in most cases. Using such policies in MaxEnt algorithms, therefore, requires constructing a tractable approximation of the mixture entropy. In this paper, we derive a simple, low-variance mixture-entropy estimator. We show that it is closely related to the sum of marginal entropies. Equipped with our entropy estimator, we derive an algorithmic variant of Soft Actor-Critic (SAC) to the mixture policy case and evaluate it on a series of continuous control tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 15 of its 45 references that are in the map. First posted on arXiv 18 Mar 2021.

Builds on

15 of 45 references.

References · 15

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2016Deep Exploration via Bootstrapped DQNcited by 518
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  9. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  10. 2017OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learningcited by 40
  11. 2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
  12. 2017Estimating Mixture Entropy with Pairwise Distancescited by 34
3 more
  1. 2018Mix&Match - Agent Curricula for Reinforcement Learningcited by 26
  2. 2019Distributional Policy Optimization: An Alternative Approach for Continuous Controlcited by 21
  3. 2020Optimistic Distributionally Robust Policy Optimizationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 0; 2026: 1.

4 papers.

  1. 2023Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraintscited by 4
  2. 2024How to Choose a Reinforcement-Learning Algorithmcited by 3
  3. 2024Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiencycited by 1
  4. 2026Revisiting Mixture Policies in Entropy-Regularized Actor-Criticcited by 1