mlsanityData to 14 Oct 2026

Evolving Reinforcement Learning Algorithms

arXiv 2101.03958 ↗First posted 8 Jan 2021cs.LG cs.AI cs.NEcited by 49

Not on the map. Why?

Abstract (arXiv)

We propose a method for meta-learning reinforcement learning algorithms by searching over the space of computational graphs which compute the loss function for a value-based model-free RL agent to optimize. The learned algorithms are domain-agnostic and can generalize to new environments not seen during training. Our method can both learn from scratch and bootstrap off known existing algorithms, like DQN, enabling interpretable modifications which improve performance. Learning from scratch on simple classical control and gridworld tasks, our method rediscovers the temporal-difference (TD) algorithm. Bootstrapped from DQN, we highlight two learned algorithms which obtain good generalization performance over other classical control tasks, gridworld type tasks, and Atari games. The analysis of the learned algorithm behavior shows resemblance to recently proposed RL algorithms that address overestimation in value-based methods.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 8 Jan 2021.

Builds on

30 of 47 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  8. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  9. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  10. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  11. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  12. 2017Progressive Neural Architecture Searchcited by 487
18 more
  1. 2016Learning to reinforcement learncited by 377
  2. 2017Large-Scale Evolution of Image Classifierscited by 348
  3. 2019The Evolved Transformercited by 180
  4. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 143
  5. 2015Neural Programmer-Interpreterscited by 101
  6. 2018Evolved Policy Gradientscited by 88
  7. 2018Evolution-Guided Policy Gradient in Reinforcement Learningcited by 83
  8. 2017Neural Optimizer Search with Reinforcement Learningcited by 82
  9. 2020Discovering Reinforcement Learning Algorithmscited by 77
  10. 2020Munchausen Reinforcement Learningcited by 69
  11. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  12. 2017Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithmcited by 65
  13. 2019Meta-Learning via Learned Losscited by 60
  14. 2020Sample-Efficient Automated Deep Reinforcement Learningcited by 32
  15. 2019Evolving Rewards to Automate Reinforcement Learningcited by 31
  16. 2020Meta-learning curiosity algorithmscited by 25
  17. 2019Learning Compositional Neural Programs with Recursive Tree Search and Planningcited by 17
  18. 2020Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategiescited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 20; 2023: 8; 2024: 10; 2025: 2; 2026: 3.

49 papers.

Sort:

By area:

  1. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  2. 2022Discovered Policy Optimisationcited by 104
  3. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  4. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  5. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  6. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 42
  7. 2022Transformers are Meta-Reinforcement Learnerscited by 41
  8. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 40
  9. 2023Evolutionary Reinforcement Learning: A Surveycited by 29
  10. 2022Context is Everything: Implicit Identification for Dynamics Adaptationcited by 27
  11. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 26
  12. 2021CARL: A Benchmark for Contextual and Adaptive Reinforcement Learningcited by 21
37 more
  1. 2022Finite Expression Method for Solving High-Dimensional Partial Differential Equationscited by 21
  2. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  3. 2022Bayesian Generational Population-Based Trainingcited by 20
  4. 2021Regularized Softmax Deep Multi-Agent Q-Learningcited by 19
  5. 2022Evolving symbolic density functionalscited by 19
  6. 2022On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learningcited by 19
  7. 2022Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimizationcited by 18
  8. 2024Discovering Temporally-Aware Reinforcement Learning Algorithmscited by 18
  9. 2023AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learningcited by 17
  10. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  11. 2023Discovering General Reinforcement Learning Algorithms with Adversarial Environment Designcited by 16
  12. 2022Reinforcement Learning with Automated Auxiliary Loss Searchcited by 13
  13. 2022A Simple Guard for Learned Optimizerscited by 9
  14. 2022Learning Symbolic Model-Agnostic Loss Functions via Meta-Learningcited by 9
  15. 2021Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRLcited by 8
  16. 2023Deep Model Predictive Optimizationcited by 7
  17. 2022Reinforcement Learning in Practice: Opportunities and Challengescited by 6
  18. 2024Behaviour Distillationcited by 5
  19. 2021CONetV2: Efficient Auto-Channel Size Optimization for CNNscited by 4
  20. 2022Lamarckian Platform: Pushing the Boundaries of Evolutionary Reinforcement Learning towards Asynchronous Commercial Gamescited by 4
  21. 2022Adaptive Prototypical Networkscited by 4
  22. 2022Hyperparameters in Contextual RL are Highly Situationalcited by 4
  23. 2024Evolutionary Reinforcement Learning: A Systematic Review and Future Directionscited by 4
  24. 2026Discovering Multiagent Learning Algorithms with Large Language Modelscited by 4
  25. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learningcited by 3
  26. 2022Learning to learn online with neuromodulated synaptic plasticity in spiking neural networkscited by 2
  27. 2022Shortest Edit Path Crossover: A Theory-driven Solution to the Permutation Problem in Evolutionary Neural Architecture Searchcited by 2
  28. 2023Unified Functional Hashing in Automatic Machine Learningcited by 2
  29. 2022Evolving Pareto-Optimal Actor-Critic Algorithms for Generalizability and Stabilitycited by 1
  30. 2024Fast and Efficient Local Search for Genetic Programming Based Loss Function Learningcited by 1
  31. 2025GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspectivecited by 1
  32. 2026Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Modelscited by 1
  33. 2021ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spacescited by 0
  34. 2023TurboGP: A flexible and advanced python based GP librarycited by 0
  35. 2024Guided Evolution with Binary Discriminators for ML Program Searchcited by 0
  36. 2024Searching Search Spaces: Meta-evolving a Geometric Encoding for Neural Networkscited by 0
  37. 2026Provably Sub-Linear Two-Timescale NeuroEvolution with Online Plasticitycited by 0