mlsanityData to 14 Oct 2026

Evolving Pareto-Optimal Actor-Critic Algorithms for Generalizability and Stability

arXiv 2204.04292 ↗First posted 8 Apr 2022cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Generalizability and stability are two key objectives for operating reinforcement learning (RL) agents in the real world. Designing RL algorithms that optimize these objectives can be a costly and painstaking process. This paper presents MetaPG, an evolutionary method for automated design of actor-critic loss functions. MetaPG explicitly optimizes for generalizability and performance, and implicitly optimizes the stability of both metrics. We initialize our loss function population with Soft Actor-Critic (SAC) and perform multi-objective optimization using fitness metrics encoding single-task performance, zero-shot generalizability to unseen environment configurations, and stability across independent runs with different random seeds. On a set of continuous control tasks from the Real-World RL Benchmark Suite, we find that our method, using a single environment during evolution, evolves algorithms that improve upon SAC's performance and generalizability by 4% and 20%, respectively, and reduce instability up to 67%. Then, we scale up to more complex environments from the Brax physics simulator and replicate generalizability tests encountered in practical settings, such as different friction coefficients. MetaPG evolves algorithms that can obtain 10% better generalizability without loss of performance within the same meta-training environment and obtain similar results to SAC when doing cross-domain evaluations in other Brax environments. The evolution results are interpretable; by analyzing the structure of the best algorithms we identify elements that help optimizing certain objectives, such as regularization terms for the critic loss.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 44 of its 62 references that are in the map. First posted on arXiv 8 Apr 2022.

Builds on

44 of 62 references.

References · 44

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2016Matching Networks for One Shot Learningcited by 2,255
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  6. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  7. 2017Deep Reinforcement Learning that Matterscited by 754
  8. 2019Solving Rubik's Cube with a Robot Handcited by 734
  9. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  10. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  11. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  12. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
32 more
  1. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  2. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  3. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  4. 2018Meta-Gradient Reinforcement Learningcited by 159
  5. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  6. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 143
  7. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  8. 2020Phasic Policy Gradientcited by 123
  9. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  10. 2018Evolved Policy Gradientscited by 88
  11. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  12. 2020Discovering Reinforcement Learning Algorithmscited by 77
  13. 2018Evolving Mario Levels in the Latent Space of a Deep Convolutional Generative Adversarial Networkcited by 76
  14. 2019Weight Agnostic Neural Networkscited by 71
  15. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  16. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  17. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  18. 2019A review on Deep Reinforcement Learning for Fluid Mechanicscited by 66
  19. 2019Meta-Learning via Learned Losscited by 60
  20. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  21. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  22. 2021Evolving Reinforcement Learning Algorithmscited by 49
  23. 2022Environment Generation for Zero-Shot Compositional Reinforcement Learningcited by 32
  24. 2019Evolving Rewards to Automate Reinforcement Learningcited by 31
  25. 2020Meta-learning curiosity algorithmscited by 25
  26. 2019On Inductive Biases in Deep Reinforcement Learningcited by 24
  27. 2019Investigating Generalisation in Continuous Deep Reinforcement Learningcited by 18
  28. 2021Emphatic Algorithms for Deep Reinforcement Learningcited by 12
  29. 2021Neuro-algorithmic Policies enable Fast Combinatorial Generalizationcited by 11
  30. 2020Reinforcement Learning Generalization with Surprise Minimizationcited by 10
  31. 2020Quantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learningcited by 5
  32. 2021Learning Synthetic Environments for Reinforcement Learning with Evolution Strategiescited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16