# Sample-Efficient Automated Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2009.01555 · HTML: https://mlsanity.com/paper/2009.01555/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2020.

## Built on by (most cited first, 30 of 32)

- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Hyperparameters in Reinforcement Learning and How To Tune Them](https://mlsanity.com/paper/2306.01324/) (2023)
- [Evolving Reinforcement Learning Algorithms](https://mlsanity.com/paper/2101.03958/) (2021)
- [Contextualize Me -- The Case for Context in Reinforcement Learning](https://mlsanity.com/paper/2202.04500/) (2022)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners](https://mlsanity.com/paper/2412.17256/) (2024)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [Bayesian Generational Population-Based Training](https://mlsanity.com/paper/2207.09405/) (2022)
- [AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learning](https://mlsanity.com/paper/2301.10339/) (2023)
- [A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning](https://mlsanity.com/paper/2412.07165/) (2024)
- [Multi-conditioned Graph Diffusion for Neural Architecture Search](https://mlsanity.com/paper/2403.06020/) (2024)
- [Reinforcement Learning with Automated Auxiliary Loss Search](https://mlsanity.com/paper/2210.06041/) (2022)
- [Automated Reinforcement Learning: An Overview](https://mlsanity.com/paper/2201.05000/) (2022)
- [Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL](https://mlsanity.com/paper/2106.15883/) (2021)
- [Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learning](https://mlsanity.com/paper/2503.01437/) (2025)
- [Towards Automatic Actor-Critic Solutions to Continuous Control](https://mlsanity.com/paper/2106.08918/) (2021)
- [Hyperparameters in Contextual RL are Highly Situational](https://mlsanity.com/paper/2212.10876/) (2022)
- [Evolutionary Reinforcement Learning: A Systematic Review and Future Directions](https://mlsanity.com/paper/2402.13296/) (2024)
- [ARLO: A Framework for Automated Reinforcement Learning](https://mlsanity.com/paper/2205.10416/) (2022)
- [Decomposing Control Lyapunov Functions for Efficient Reinforcement Learning](https://mlsanity.com/paper/2403.12210/) (2024)
- [Adaptive Q-Network: On-the-fly Target Selection for Deep Reinforcement Learning](https://mlsanity.com/paper/2405.16195/) (2024)
- [ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning](https://mlsanity.com/paper/2409.18827/) (2024)
- [Towards robust and domain agnostic reinforcement learning competitions](https://mlsanity.com/paper/2106.03748/) (2021)
- [Shrink-Perturb Improves Architecture Mixing during Population Based Training for Neural Architecture Search](https://mlsanity.com/paper/2307.15621/) (2023)
- [Skill-aware Mutual Information Optimisation for Generalisation in Reinforcement Learning](https://mlsanity.com/paper/2406.04815/) (2024)
- [Multiple-Frequencies Population-Based Training](https://mlsanity.com/paper/2506.03225/) (2025)
- [Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies](https://mlsanity.com/paper/2508.03194/) (2025)
- [REIN-2: Giving Birth to Prepared Reinforcement Learning Agents Using Reinforcement Learning Agents](https://mlsanity.com/paper/2110.05128/) (2021)
- [Automatic tuning of hyper-parameters of reinforcement learning algorithms using Bayesian optimization with behavioral cloning](https://mlsanity.com/paper/2112.08094/) (2021)

## Abstract (arXiv)

Despite significant progress in challenging problems across various domains, applying state-of-the-art deep reinforcement learning (RL) algorithms remains challenging due to their sensitivity to the choice of hyperparameters. This sensitivity can partly be attributed to the non-stationarity of the RL problem, potentially requiring different hyperparameter settings at various stages of the learning process. Additionally, in the RL setting, hyperparameter optimization (HPO) requires a large number of environment interactions, hindering the transfer of the successes in RL to real-world applications. In this work, we tackle the issues of sample-efficient and dynamic HPO in RL. We propose a population-based automated RL (AutoRL) framework to meta-optimize arbitrary off-policy RL algorithms. In this framework, we optimize the hyperparameters and also the neural architecture while simultaneously training the agent. By sharing the collected experience across the population, we substantially increase the sample efficiency of the meta-optimization. We demonstrate the capabilities of our sample-efficient AutoRL approach in a case study with the popular TD3 algorithm in the MuJoCo benchmark suite, where we reduce the number of environment interactions needed for meta-optimization by up to an order of magnitude compared to population-based training.