mlsanityData to 14 Oct 2026

Towards Automatic Actor-Critic Solutions to Continuous Control

arXiv 2106.08918 ↗First posted 16 Jun 2021cs.LG cs.NE cs.SYcited by 5

Not on the map. Why?

Abstract (arXiv)

Model-free off-policy actor-critic methods are an efficient solution to complex continuous control tasks. However, these algorithms rely on a number of design tricks and hyperparameters, making their application to new domains difficult and computationally expensive. This paper creates an evolutionary approach that automatically tunes these design decisions and eliminates the RL-specific hyperparameters from the Soft Actor-Critic algorithm. Our design is sample efficient and provides practical advantages over baseline approaches, including improved exploration, generalization over multiple control frequencies, and a robust ensemble of high-performance policies. Empirically, we show that our agent outperforms well-tuned hyperparameter settings in popular benchmarks from the DeepMind Control Suite. We then apply it to less common control tasks outside of simulated robotics to find high-performance solutions with minimal compute and research effort.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

42 of 56 references.

References · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  9. 2018Exploration by Random Network Distillationcited by 993
  10. 2018DeepMind Control Suitecited by 956
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  12. 2017Deep Reinforcement Learning that Matterscited by 754
30 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  2. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  3. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  4. 2019AutoML: A Survey of the State-of-the-Artcited by 390
  5. 2017Population Based Training of Neural Networkscited by 367
  6. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  7. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  8. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  9. 2020Revisiting Fundamentals of Experience Replaycited by 175
  10. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  11. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  12. 2017Time Limits in Reinforcement Learningcited by 79
  13. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  14. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  15. 2019Making Deep Q-learning methods robust to time discretizationcited by 58
  16. 2020OR-Gym: A Reinforcement Learning Library for Operations Research Problemscited by 51
  17. 2020Discount Factor as a Regularizer in Reinforcement Learningcited by 45
  18. 2020Sample-Efficient Automated Deep Reinforcement Learningcited by 32
  19. 2020Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learningcited by 28
  20. 2019If MaxEnt RL is the Answer, What is the Question?cited by 27
  21. 2020Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learningcited by 27
  22. 2019On Inductive Biases in Deep Reinforcement Learningcited by 24
  23. 2019ORL: Reinforcement Learning Benchmarks for Online Stochastic Optimization Problemscited by 21
  24. 2020GRAC: Self-Guided and Self-Regularized Actor-Criticcited by 21
  25. 2021An Analysis of Frame-skipping in Reinforcement Learningcited by 18
  26. 2020Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradientcited by 14
  27. 2020Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategiescited by 11
  28. 2017Online Meta-learning by Parallel Algorithm Competitioncited by 9
  29. 2019Fast Efficient Hyperparameter Tuning for Policy Gradientscited by 6
  30. 2016Introduction to the "Industrial Benchmark"cited by 3

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 0; 2024: 2; 2025: 1.

5 papers.

  1. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 26
  2. 2022Combining Evolution and Deep Reinforcement Learning for Policy Search: a Surveycited by 23
  3. 2024When to Sense and Control? A Time-adaptive Approach for Continuous-Time RLcited by 9
  4. 2022Simultaneously Updating All Persistence Values in Reinforcement Learningcited by 2
  5. 2025EvoRL: A GPU-accelerated Framework for Evolutionary Reinforcement Learningcited by 1