mlsanityData to 7 Oct 2026

Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning

arXiv 2501.17827 ↗First posted 29 Jan 2025cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Existing actor-critic algorithms, which are popular for continuous control reinforcement learning (RL) tasks, suffer from poor sample efficiency due to lack of principled exploration mechanism within them. Motivated by the success of Thompson sampling for efficient exploration in RL, we propose a novel model-free RL algorithm, Langevin Soft Actor Critic (LSAC), which prioritizes enhancing critic learning through uncertainty estimation over policy optimization. LSAC employs three key innovations: approximate Thompson sampling through distributional Langevin Monte Carlo (LMC) based Q updates, parallel tempering for exploring multiple modes of the posterior of the Q function, and diffusion synthesized state-action samples regularized with Q action gradients. Our extensive experiments demonstrate that LSAC outperforms or matches the performance of mainstream model-free RL algorithms for continuous control tasks. Notably, LSAC marks the first successful application of an LMC based Thompson sampling in continuous control tasks with continuous action spaces.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 49 of its 71 references that are in the map. First posted on arXiv 29 Jan 2025.

Builds on

49 of 71 references.

References · 49

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  10. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  12. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
37 more
  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  2. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  3. 2019An Introduction to Variational Autoencoderscited by 997
  4. 2018DeepMind Control Suitecited by 935
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  7. 2016Deep Exploration via Bootstrapped DQNcited by 518
  8. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  10. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  11. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  12. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  13. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  14. 2017Noisy Networks for Explorationcited by 262
  15. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  16. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  17. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  18. 2017Parameter Space Noise for Explorationcited by 158
  19. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 126
  20. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  21. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
  22. 2019Frequentist Regret Bounds for Randomized Least-Squares Value Iterationcited by 100
  23. 2023Synthetic Experience Replaycited by 100
  24. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  25. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 81
  26. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
  27. 2023Score Regularized Policy Optimization through Diffusion Behaviorcited by 67
  28. 2019Worst-Case Regret Bounds for Exploration via Randomized Value Functionscited by 57
  29. 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
  30. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
  31. 2023Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlocited by 29
  32. 2018Langevin-gradient parallel tempering for Bayesian neural learningcited by 14
  33. 2021Near-Optimal Randomized Exploration for Tabular Markov Decision Processescited by 14
  34. 2024More Efficient Randomized Exploration for Reinforcement Learning via Approximate Samplingcited by 12
  35. 2021An Introduction to Hamiltonian Monte Carlo Method for Samplingcited by 8
  36. 2017Enhanced Experience Replay Generation for Efficient Reinforcement Learningcited by 6
  37. 2023Dynamic Update-to-Data Ratio: Minimizing World Model Overfittingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 9.

17 papers.

Sort:

By area:

  1. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 52
  2. 2026Maximum Likelihood Reinforcement Learningcited by 29
  3. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policiescited by 7
  4. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  5. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  6. 2025Exploratory Diffusion Model for Unsupervised Reinforcement Learningcited by 4
  7. 2025Optimistic critics can empower small actorscited by 4
  8. 2026Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flowcited by 4
  9. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
  10. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 3
  11. 2025Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularizationcited by 3
  12. 2026Safe Langevin Soft Actor Criticcited by 1
5 more
  1. 2026Direct Soft-Policy Sampling via Langevin Dynamicscited by 1
  2. 2025Predictive Spike Timing Enables Distributed Shortest Path Computation in Spiking Neural Networkscited by 0
  3. 2025An Information-Theoretic Analysis of OOD Generalization in Meta-Reinforcement Learningcited by 0
  4. 2026Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distributioncited by 0
  5. 2026Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descentcited by 0