Data to 7 Oct 2026

Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents

arXiv 1709.06009 ↗First posted 18 Sep 2017cs.LGcited by 267

Not on the map. Why?

Cited by 20 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The Arcade Learning Environment (ALE) is an evaluation platform that poses the challenge of building AI agents with general competency across dozens of Atari 2600 games. It supports a variety of different problem settings and it has been receiving increasing attention from the scientific community, leading to some high-profile success stories such as the much publicized Deep Q-Networks (DQN). In this article we take a big picture look at how the ALE is being used by the research community. We show how diverse the evaluation methodologies in the ALE have become with time, and highlight some key concerns when evaluating agents in the ALE. We use this discussion to present some methodological best practices and provide new benchmark results using these best practices. To further the progress in the field, we introduce a new version of the ALE that supports multiple game modes and provides a form of stochasticity we call sticky actions. We conclude this big picture look by revisiting challenges posed when the ALE was introduced, summarizing the state-of-the-art in various problems and highlighting problems that remain open.

Key facts. Cited by 267 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Sep 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 3; 2020: 3; 2021: 61; 2022: 43; 2023: 43; 2024: 48; 2025: 40; 2026: 26.

267 papers.

Sort:

By area:

2023Mastering Diverse Domains through World Modelscited by 1,193
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
2021The Benchmark Lotterycited by 133
2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 97
2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90

38 more

2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 80
2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 62
2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
2023Empirical Design in Reinforcement Learningcited by 54
2021Muesli: Combining Improvements in Policy Optimizationcited by 53
2021Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learningcited by 53
2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
2021Nondeterminism and Instability in Neural Network Optimizationcited by 47
2021Vector Quantized Models for Planningcited by 47
2021Bootstrapped Meta-Learningcited by 45
2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
2025Pre-Trained Video Generative Models as World Simulatorscited by 43
2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 42
2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 41
2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
2021CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agentscited by 36
2022A Review of Uncertainty for Deep Reinforcement Learningcited by 36
2023Small batch deep reinforcement learningcited by 35
2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 32
2023Stochastic Generative Flow Networkscited by 32
2022Chain of Thought Imitation with Procedure Cloningcited by 30

50 most cited of 267 All 267 (JSON)