mlsanityData to 14 Oct 2026

How Should We Meta-Learn Reinforcement Learning Algorithms?

arXiv 2507.17668 ↗First posted 23 Jul 2025cs.LG cs.AIcited by 5

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

The process of meta-learning algorithms from data, instead of relying on manual design, is growing in popularity as a paradigm for improving the performance of machine learning systems. Meta-learning shows particular promise for reinforcement learning (RL), where algorithms are often adapted from supervised or unsupervised learning despite their suboptimality for RL. However, until now there has been a severe lack of comparison between different meta-learning algorithms, such as using evolution to optimise over black-box functions or LLMs to propose code. In this paper, we carry out this empirical comparison of the different approaches when applied to a range of meta-learned algorithms which target different parts of the RL pipeline. In addition to meta-train and meta-test performance, we also investigate factors including the interpretability, sample cost and train time for each meta-learning algorithm. Based on these findings, we propose several guidelines for meta-learning new RL algorithms which will help ensure that future learned algorithms are as performant as possible.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 64 of its 99 references that are in the map. First posted on arXiv 23 Jul 2025.

Builds on

64 of 99 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planning26 Feb
  2. 2024Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps22 Dec
  3. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization16 Dec
  4. 2024A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning10 Dec
  5. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn7 Sep
  6. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?9 Jul
  7. 2024Discovering Temporally-Aware Reinforcement Learning Algorithms8 Feb

Other references · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2018Exploration by Random Network Distillationcited by 993
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2016Learning to learn by gradient descent by gradient descentcited by 620
  9. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  10. 2018Born Again Neural Networkscited by 546
  11. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  12. 2023Symbolic Discovery of Optimization Algorithmscited by 434
38 more
  1. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  2. 2024Automated Design of Agentic Systemscited by 407
  3. 2020Discovering Symbolic Models from Deep Learning with Inductive Biasescited by 344
  4. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  5. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  6. 2017Noisy Networks for Explorationcited by 266
  7. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
  8. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  9. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  10. 2023Understanding plasticity in neural networkscited by 174
  11. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
  12. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
  13. 2017Parameter Space Noise for Explorationcited by 161
  14. 2022Evolution through Large Modelscited by 134
  15. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  16. 2022Discovered Policy Optimisationcited by 104
  17. 2018Tunability: Importance of Hyperparameters of Machine Learning Algorithmscited by 99
  18. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  19. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 88
  20. 2021Gradients are Not All You Needcited by 86
  21. 2020Discovering Reinforcement Learning Algorithmscited by 77
  22. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  23. 2024OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Codecited by 74
  24. 2020Self-Distillation as Instance-Specific Label Smoothingcited by 71
  25. 2022Rediscovering orbital mechanics with machine learningcited by 71
  26. 2017Boltzmann Exploration Done Rightcited by 69
  27. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 69
  28. 2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
  29. 2020Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselvescited by 58
  30. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  31. 2020Meta Learning Backpropagation And Improving Itcited by 49
  32. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  33. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 42
  34. 2022evosax: JAX-based Evolution Strategiescited by 41
  35. 2018Meta-Learning Update Rules for Unsupervised Representation Learningcited by 37
  36. 2023Small batch deep reinforcement learningcited by 36
  37. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  38. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 28

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Evolution Strategies at the Hyperscalecited by 25
  2. 2026AI Research Preference Modelscited by 10
  3. 2026Simple Baselines are Competitive with Code Evolutioncited by 9
  4. 2025On Discovering Algorithms for Adversarial Imitation Learningcited by 1
  5. 2026PyVRP^+: LLM-Driven Metacognitive Heuristic Evolution for Hybrid Genetic Search in Vehicle Routing Problemscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Theoretical Analysis of Meta Reinforcement Learning: Generalization Bounds and Convergence Guarantees
  2. 2021Offline Meta-Reinforcement Learning with Online Self-Supervision
  3. 2024Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator
  4. 2024MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning
  5. 2024Black Box Meta-Learning Intrinsic Rewards
  6. 2020Offline Meta-Reinforcement Learning with Advantage Weighting
  7. 2021Offline Meta-Reinforcement Learning for Industrial Insertion
  8. 2026Meta-Learning and Meta-Reinforcement Learning -- Tracing the Path towards DeepMind's Adaptive Agent