Mastering Atari 2600 Games with Discovered Options

arXiv 2610.03604 ↗First posted 2 Oct 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization also fits: Deep RL algorithms and representations

Abstract (arXiv)

Temporal abstractions, often instantiated as options, have long been regarded as a mechanism for accelerating credit assignment, facilitating exploration, and enabling generalisation in reinforcement learning (RL). However, developing general option discovery methods that are effective in large-scale, high-dimensional domains remains a fundamental challenge. Existing option discovery methods are either confined to relatively simple domains, depend on handcrafted or quasi-symbolic representations, or offer little improvement over learning without options. We present Wayfarer, a general, domain-agnostic, online deep RL agent that discovers options through Laplacian representation learning from high-dimensional observations and leverages them for control. We show that the resulting options simultaneously improve exploration, accelerate credit assignment, and generalise effectively to unseen settings, enabling substantially faster learning of complex policies. Wayfarer achieves state-of-the-art performance among single-stream agents on the most challenging Atari 2600 games, with the largest gains in games that require long-horizon exploration and strategic behaviour, such as Montezuma's Revenge and Private Eye.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 45 of its 94 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

45 of 94 references.

References · 45

2016Layer Normalizationcited by 5,669
2013Playing Atari with Deep Reinforcement Learningcited by 3,274
2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
2015Prioritized Experience Replaycited by 1,099
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
2017A Distributional Perspective on Reinforcement Learningcited by 765
2016The Option-Critic Architecturecited by 448
2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337

33 more

2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
2016Variational Intrinsic Controlcited by 267
2017Noisy Networks for Explorationcited by 266
2020Never Give Up: Learning Directed Exploration Strategiescited by 212
2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 110
2017Eigenoption Discovery through the Deep Successor Representationcited by 86
2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 86
2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
2020Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Surveycited by 85
2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
2024Simplifying Deep Temporal Difference Learningcited by 78
2024Normalization and effective learning rates in reinforcement learningcited by 58
2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 54
2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 45
2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 44
2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 42
2019Discovering Options for Exploration by Minimizing Cover Timecited by 31
2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 29
2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 28
2021Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawingcited by 23
2025Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learningcited by 22
2024Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learningcited by 21
2022Reward-Respecting Subtasks for Model-Based Reinforcement Learningcited by 16
2022Agent-Controller Representations: Principled Offline RL with Rich Exogenous Informationcited by 16
2023Proper Laplacian Representation Learningcited by 11
2022Temporal Abstractions-Augmented Temporally Contrastive Learning: An Alternative to the Laplacian in RLcited by 7
2026Laplacian Representations for Decision-Time Planningcited by 5
2025Reward-Aware Proto-Representations in Reinforcement Learningcited by 4
2025Spectral Bellman Method: Unifying Representation and Exploration in RLcited by 4
2026The Laplacian Keyboard: Beyond the Linear Spancited by 1
2026Learning to Perceive the World Through Control: Empowerment-Based Representation Learningcited by 1
2026Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discoverycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way