mlsanityData to 7 Oct 2026

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

arXiv 1801.01290 ↗First posted 4 Jan 2018cs.LG cs.AI stat.MLcited by 4,635

Not on the map. Why?

Cited by 669 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.

Key facts. Cited by 4,635 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 3; 2019: 15; 2020: 28; 2021: 619; 2022: 712; 2023: 745; 2024: 831; 2025: 889; 2026: 793.

4,635 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2023Mastering Diverse Domains through World Modelscited by 1,193
  5. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  9. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  12. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
38 more
  1. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  2. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  3. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  4. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  5. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  6. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
  7. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  8. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  9. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  10. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  11. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  12. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  13. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  14. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  15. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  16. 2026Reinforcement Learning via Self-Distillationcited by 285
  17. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  18. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 281
  19. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  20. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  21. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  22. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  23. 2021How to Train Your Energy-Based Modelscited by 271
  24. 2025Learning to Reason under Off-Policy Guidancecited by 266
  25. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  26. 2024Diffusion Policy Policy Optimizationcited by 255
  27. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 244
  28. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  29. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  30. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  31. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  32. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  33. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  34. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  35. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  36. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  37. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  38. 2022Multi-Task Learning as a Bargaining Gamecited by 209

50 most cited of 4,635 All 4,635 (JSON)