mlsanityData to 14 Oct 2026

Revisiting Mixture Policies in Entropy-Regularized Actor-Critic

arXiv 2605.09157 ↗First posted 9 May 2026cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Mixture policies theoretically offer greater flexibility than unimodal policies in continuous action reinforcement learning, but the practical benefits of this complexity remain elusive. Mixture policies are notably absent from most state-of-the-art algorithms, raising a fundamental question: Is the added representational overhead useful? We show that increased flexibility can theoretically enhance solution quality and entropy robustness. Yet standard algorithms like SAC do not leverage these advantages. A core issue is the lack of a low-variance reparameterization trick for mixtures, a luxury Gaussian policies enjoy. We propose a marginalized reparameterization (MRP) estimator to address this, proving it offers lower variance than the standard likelihood-ratio (LR) approach. Our experiments across Gym MuJoCo, DeepMind Control Suite, and MetaWorld show that MRP mixture policies significantly outperform their LR ones, and reach parity (sometimes better) with Gaussian counterparts. In addition, we do find several cases where MRP mixture policies exhibit clear empirical advantages. In this paper, we provide a clearer understanding of the trade-offs involved, elevating MRP mixture policies from theoretical curiosity to a practical tool.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 36 of its 52 references that are in the map. First posted on arXiv 9 May 2026.

Builds on

36 of 52 references.

References · 36

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  11. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  12. 2018DeepMind Control Suitecited by 956
24 more
  1. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  2. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  3. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  4. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  5. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  6. 2018Understanding the impact of entropy on policy optimizationcited by 176
  7. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  8. 2022MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlcited by 80
  9. 2021Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learningcited by 50
  10. 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 46
  11. 2018Variance reduction properties of the reparameterization trickcited by 41
  12. 2017Reducing Reparameterization Gradient Variancecited by 27
  13. 2021Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learningcited by 27
  14. 2021Specializing Versatile Skill Libraries using Local Mixture of Expertscited by 24
  15. 2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Criticcited by 18
  16. 2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
  17. 2018Greedy Actor-Critic: A New Conditional Cross-Entropy Method for Policy Improvementcited by 12
  18. 2021On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Controlcited by 12
  19. 2021Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Modelscited by 12
  20. 2021Model-free Policy Learning with Reward Gradientscited by 9
  21. 2023Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithmscited by 7
  22. 2018Implicit Policy for Reinforcement Learningcited by 6
  23. 2020Off-policy Maximum Entropy Reinforcement Learning : Soft Actor-Critic with Advantage Weighted Mixture Policy(SAC-AWMP)cited by 5
  24. 2021Maximum Entropy Reinforcement Learning with Mixture Policiescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitionerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Refined Analysis of Entropy-Regularized Actor-Critic
  2. 2024Diffusion Actor-Critic with Entropy Regulator
  3. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization
  4. 2026FERPO: Forward Entropy-Regularized Policy Optimization
  5. 2025State Entropy Regularization for Robust Reinforcement Learning
  6. 2026Unregularized Convergence of Single-Loop, Entropy-Regularized Natural Actor-Critic
  7. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
  8. 2026Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods