# Learnable Behavior Control: Breaking Atari Human World Records via Sample-Efficient Behavior Selection

arXiv: https://arxiv.org/abs/2305.05239 · HTML: https://mlsanity.com/paper/2305.05239/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Safe and Efficient Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1606.02647/) (2016)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [Effective Diversity in Population Based Reinforcement Learning](https://mlsanity.com/paper/2002.00632/) (2020)
- [Going Beyond Linear Transformers with Recurrent Fast Weight Programmers](https://mlsanity.com/paper/2106.06295/) (2021)
- [Category-Level 6D Object Pose Estimation via Cascaded Relation and Recurrent Reconstruction Networks](https://mlsanity.com/paper/2108.08755/) (2021)
- [ESCM2: Entire Space Counterfactual Multi-Task Model for Post-Click Conversion Rate Estimation](https://mlsanity.com/paper/2204.05125/) (2022)
- [Off-Policy Actor-Critic with Shared Experience Replay](https://mlsanity.com/paper/1909.11583/) (2019)
- [Human-level Atari 200x faster](https://mlsanity.com/paper/2209.07550/) (2022)
- [Disentangled Contrastive Learning for Social Recommendation](https://mlsanity.com/paper/2208.08723/) (2022)
- [LHNN: Lattice Hypergraph Neural Network for VLSI Congestion Prediction](https://mlsanity.com/paper/2203.12831/) (2022)
- [A Review for Deep Reinforcement Learning in Atari:Benchmarks, Challenges, and Solutions](https://mlsanity.com/paper/2112.04145/) (2021)
- [Generalized Data Distribution Iteration](https://mlsanity.com/paper/2206.03192/) (2022)
- [An Entropy Regularization Free Mechanism for Policy-based Reinforcement Learning](https://mlsanity.com/paper/2106.00707/) (2021)
- [Critic PI2: Master Continuous Planning via Policy Improvement with Path Integrals and Deep Actor-Critic Reinforcement Learning](https://mlsanity.com/paper/2011.06752/) (2020)
- [DR-Label: Improving GNN Models for Catalysis Systems by Label Deconstruction and Reconstruction](https://mlsanity.com/paper/2303.02875/) (2023)

## Built on by (most cited first, 14 of 14)

- [Mixture of Experts in a Mixture of RL settings](https://mlsanity.com/paper/2406.18420/) (2024)
- [DistDF: Time-Series Forecasting Needs Joint-Distribution Wasserstein Alignment](https://mlsanity.com/paper/2510.24574/) (2025)
- [PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inference](https://mlsanity.com/paper/2407.05010/) (2024)
- [Quadratic Direct Forecast for Training Multi-Step Time-Series Forecast Models](https://mlsanity.com/paper/2511.00053/) (2025)
- [CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks](https://mlsanity.com/paper/2603.18736/) (2026)
- [RAGRouter-Bench: A Dataset and Benchmark for Adaptive RAG Routing](https://mlsanity.com/paper/2602.00296/) (2026)
- [Variational Supervised Contrastive Learning](https://mlsanity.com/paper/2506.07413/) (2025)
- [Fine-tuning Flow Matching Generative Models with Intermediate Feedback](https://mlsanity.com/paper/2510.18072/) (2025)
- [Modeling Task Relationships in Multi-variate Soft Sensor with Balanced Mixture-of-Experts](https://mlsanity.com/paper/2305.16360/) (2023)
- [β-DQN: Improving Deep Q-Learning By Evolving the Behavior](https://mlsanity.com/paper/2501.00913/) (2025)
- [Perceptual Similarity for Measuring Decision-Making Style and Policy Diversity in Games](https://mlsanity.com/paper/2408.06051/) (2024)
- [SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning](https://mlsanity.com/paper/2608.26432/) (2026)
- [One is More: Diverse Perspectives within a Single Network for Efficient DRL](https://mlsanity.com/paper/2310.14009/) (2023)
- [Can We Read the Mind of an Audio LLM? A Verbalizable, Multilingual Middle-Layer Workspace](https://mlsanity.com/paper/2608.24958/) (2026)

## Abstract (arXiv)

The exploration problem is one of the main challenges in deep reinforcement learning (RL). Recent promising works tried to handle the problem with population-based methods, which collect samples with diverse behaviors derived from a population of different exploratory policies. Adaptive policy selection has been adopted for behavior control. However, the behavior selection space is largely limited by the predefined policy population, which further limits behavior diversity. In this paper, we propose a general framework called Learnable Behavioral Control (LBC) to address the limitation, which a) enables a significantly enlarged behavior selection space via formulating a hybrid behavior mapping from all policies; b) constructs a unified learnable process for behavior selection. We introduce LBC into distributed off-policy actor-critic methods and achieve behavior control via optimizing the selection of the behavior mappings with bandit-based meta-controllers. Our agents have achieved 10077.52% mean human normalized score and surpassed 24 human world records within 1B training frames in the Arcade Learning Environment, which demonstrates our significant state-of-the-art (SOTA) performance without degrading the sample efficiency.