mlsanityData to 7 Oct 2026

BiERL: A Meta Evolutionary Reinforcement Learning Framework via Bilevel Optimization

arXiv 2308.01207 ↗First posted 1 Aug 2023cs.NE cs.AI cs.LGcited by 0

On the map: Synthetic data and self-improvement for LLMs → Self-evolving LLM agents

Abstract (arXiv)

Evolutionary reinforcement learning (ERL) algorithms recently raise attention in tackling complex reinforcement learning (RL) problems due to high parallelism, while they are prone to insufficient exploration or model collapse without carefully tuning hyperparameters (aka meta-parameters). In the paper, we propose a general meta ERL framework via bilevel optimization (BiERL) to jointly update hyperparameters in parallel to training the ERL model within a single agent, which relieves the need for prior domain knowledge or costly optimization procedure before model deployment. We design an elegant meta-level architecture that embeds the inner-level's evolving experience into an informative population representation and introduce a simple and feasible evaluation of the meta-level fitness function to facilitate learning efficiency. We perform extensive experiments in MuJoCo and Box2D tasks to verify that as a general framework, BiERL outperforms various baselines and consistently improves the learning performance for a diversity of ERL algorithms.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Self-evolving LLM agents, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 30 of its 42 references that are in the map. First posted on arXiv 1 Aug 2023.

Builds on

30 of 42 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2016OpenAI Gymcited by 1,684
  4. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  5. 2017Population Based Training of Neural Networkscited by 351
  6. 2017Parameter Space Noise for Explorationcited by 158
  7. 2017A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applicationscited by 126
  8. 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
  9. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  10. 2019Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functionscited by 104
  11. 2018Evolved Policy Gradientscited by 86
  12. 2019Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAScited by 83
18 more
  1. 2020Meta-Learning with Adaptive Hyperparameterscited by 79
  2. 2018Evolution-Guided Policy Gradient in Reinforcement Learningcited by 78
  3. 2020Provably Efficient Online Hyperparameter Optimization with Population-Based Banditscited by 61
  4. 2019ES-MAML: Simple Hessian-Free Meta Learningcited by 57
  5. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimizationcited by 41
  6. 2019Collaborative Evolutionary Reinforcement Learningcited by 36
  7. 2020Lifelong Incremental Reinforcement Learning with Online Bayesian Inferencecited by 23
  8. 2019Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordinationcited by 14
  9. 2021Optimizing Large-Scale Hyperparameters via Automated Learning Algorithmcited by 13
  10. 2022ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representationcited by 11
  11. 2018Meta-Learning by the Baldwin Effectcited by 10
  12. 2017Online Meta-learning by Parallel Algorithm Competitioncited by 9
  13. 2020Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategiescited by 9
  14. 2021Rule-Based Reinforcement Learning for Efficient Robot Navigation with Space Reductioncited by 9
  15. 2021EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimizationcited by 9
  16. 2019Evolvability ES: Scalable and Direct Optimization of Evolvabilitycited by 7
  17. 2020Instance Weighted Incremental Evolution Strategies for Reinforcement Learning in Dynamic Environmentscited by 4
  18. 2021Online Hyperparameter Meta-Learning with Hypergradient Distillationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback
  2. 2025Differentiable Evolutionary Reinforcement Learning
  3. 2026EVOM: Agentic Meta-Evolution of Actor-Critic Architectures for Reinforcement Learning
  4. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF
  5. 2026Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning
  6. 2026Hypergradient-based Bilevel Reinforcement Learning with Improved Sample Complexity
  7. 2025Utilizing Evolution Strategies to Train Transformers in Reinforcement Learning
  8. 2026RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms