mlsanityData to 14 Oct 2026

S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic

arXiv 2405.00987 ↗First posted 2 May 2024cs.LGcited by 18

On the map: Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

Learning expressive stochastic policies instead of deterministic ones has been proposed to achieve better stability, sample complexity, and robustness. Notably, in Maximum Entropy Reinforcement Learning (MaxEnt RL), the policy is modeled as an expressive Energy-Based Model (EBM) over the Q-values. However, this formulation requires the estimation of the entropy of such EBMs, which is an open problem. To address this, previous MaxEnt RL methods either implicitly estimate the entropy, resulting in high computational complexity and variance (SQL), or follow a variational inference procedure that fits simplified actor distributions (e.g., Gaussian) for tractability (SAC). We propose Stein Soft Actor-Critic (S2AC), a MaxEnt RL algorithm that learns expressive policies without compromising efficiency. Specifically, S2AC uses parameterized Stein Variational Gradient Descent (SVGD) as the underlying policy. We derive a closed-form expression of the entropy of such policies. Our formula is computationally efficient and only depends on first-order derivatives and vector products. Empirical results show that S2AC yields more optimal solutions to the MaxEnt objective than SQL and SAC in the multi-goal environment, and outperforms SAC and SQL on the MuJoCo benchmark. Our code is available at: https://github.com/SafaMessaoud/S2AC-Energy-Based-RL-with-Stein-Soft-Actor-Critic

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 43 of its 99 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

43 of 99 references.

References · 43

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2015Variational Inference with Normalizing Flowscited by 2,170
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  9. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 507
  10. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  11. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  12. 2018Maximum a Posteriori Policy Optimisationcited by 328
31 more
  1. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 176
  2. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  3. 2016A Theory of Generative ConvNetcited by 174
  4. 2017Stein Variational Gradient Descent as Gradient Flowcited by 152
  5. 2020Learning Energy-Based Models by Diffusion Recovery Likelihoodcited by 123
  6. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 81
  7. 2020Munchausen Reinforcement Learningcited by 69
  8. 2016Cooperative Training of Descriptor and Generator Networkscited by 60
  9. 2021Trajectory Prediction with Latent Belief Energy-Based Modelcited by 60
  10. 2017Generalizing Hamiltonian Monte Carlo with Neural Networkscited by 52
  11. 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 46
  12. 2017Stein Variational Policy Gradientcited by 44
  13. 2022A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Modelcited by 44
  14. 2020Learning Energy-Based Model with Variational Auto-Encoder as Amortized Samplercited by 43
  15. 2019Exponential Family Estimation via Adversarial Dynamics Embeddingcited by 30
  16. 2017Learning to Draw Samples with Amortized Stein Variational Gradient Descentcited by 28
  17. 2020Generative VoxelNet: Learning Energy-Based Models for 3D Shape Synthesis and Analysiscited by 26
  18. 2021Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problemscited by 26
  19. 2016Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learningcited by 21
  20. 2018Meta-Learning for Stochastic Gradient MCMCcited by 16
  21. 2018Kernel Exponential Family Estimation via Doubly Dual Embeddingcited by 16
  22. 2018Structural Consistency and Controllability for Diverse Colorizationcited by 15
  23. 2022Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guaranteescited by 14
  24. 2017Calibrating Energy-based Generative Adversarial Networkscited by 13
  25. 2020Iterative Amortized Policy Optimizationcited by 13
  26. 2018Boosting Trust Region Policy Optimization by Normalizing Flows Policycited by 11
  27. 2021Learning Cycle-Consistent Cooperative Networks via Alternating MCMC Teaching for Unsupervised Cross-Domain Translationcited by 8
  28. 2022Stein Variational Goal Generation for adaptive Exploration in Multi-Goal Reinforcement Learningcited by 6
  29. 2022Combating Mode Collapse in GANs via Manifold Entropy Estimationcited by 4
  30. 2020Can We Learn Heuristics For Graphical Model Inference Using Reinforcement Learning?cited by 2
  31. 2022Policy Gradient With Serial Markov Chain Reasoningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 2.

18 papers.

Sort:

By area:

  1. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 58
  2. 2024Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flowcited by 26
  3. 2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
  4. 2025Wasserstein Policy Optimizationcited by 13
  5. 2025Enhanced DACER Algorithm with High Diffusion Efficiencycited by 13
  6. 2025Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoningcited by 6
  7. 2024Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Spacecited by 4
  8. 2024Long-time asymptotics of noisy SVGD outside the population limitcited by 3
  9. 2025DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertaintycited by 3
  10. 2025Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularizationcited by 3
  11. 2025Offline Reinforcement Learning with Generative Trajectory Policiescited by 1
  12. 2025A Theoretical Lens for RL-Tuned Language Models via Energy-Based Modelscited by 1
6 more
  1. 2026Revisiting Mixture Policies in Entropy-Regularized Actor-Criticcited by 1
  2. 2024Edge Delayed Deep Deterministic Policy Gradient: efficient continuous control for edge scenarioscited by 0
  3. 2025Distributions as Actions: A Unified Framework for Diverse Action Spacescited by 0
  4. 2025Q-Guided Stein Variational Model Predictive Control via RL-informed Policy Priorcited by 0
  5. 2025Adaptive Kernel Selection for Stein Variational Gradient Descentcited by 0
  6. 2026Flow Perturbation++: Multi-Step Unbiased Jacobian Estimation for High-Dimensional Boltzmann Samplingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Max-Entropy Reinforcement Learning with Flow Matching and A Case Study on LQR
  2. 2025Maximum Entropy Reinforcement Learning with Diffusion Policy
  3. 2025Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning
  4. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
  5. 2024Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimation
  6. 2025Effective Reinforcement Learning Control using Conservative Soft Actor-Critic
  7. 2026Generative Actor-Critic with Soft Bridge Policies
  8. 2026Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matching