mlsanityData to 7 Oct 2026

When should agents explore?

arXiv 2108.11811 ↗First posted 26 Aug 2021cs.LG cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Exploration remains a central challenge for reinforcement learning (RL). Virtually all existing methods share the feature of a monolithic behaviour policy that changes only gradually (at best). In contrast, the exploratory behaviours of animals and humans exhibit a rich diversity, namely including forms of switching between modes. This paper presents an initial study of mode-switching, non-monolithic exploration for RL. We investigate different modes to switch between, at what timescales it makes sense to switch, and what signals make for good switching triggers. We also propose practical algorithmic components that make the switching mechanism adaptive and robust, which enables flexibility without an accompanying hyper-parameter-tuning burden. Finally, we report a promising and detailed analysis on Atari, using two-mode exploration and switching at sub-episodic time-scales.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 26 of its 78 references that are in the map. First posted on arXiv 26 Aug 2021.

Builds on

26 of 78 references.

References · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Prioritized Experience Replaycited by 1,070
  3. 2018Exploration by Random Network Distillationcited by 966
  4. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  5. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  6. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  7. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  8. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  9. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  10. 2016Variational Intrinsic Controlcited by 262
  11. 2018Distributed Prioritized Experience Replaycited by 260
  12. 2017Count-Based Exploration with Neural Density Modelscited by 255
14 more
  1. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  2. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  3. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  4. 2018Universal Successor Features Approximatorscited by 102
  5. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 65
  6. 2019A Bayesian Approach to Robust Reinforcement Learningcited by 32
  7. 2019Discovering Options for Exploration by Minimizing Cover Timecited by 30
  8. 2020Temporally-Extended ε-Greedy Explorationcited by 29
  9. 2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
  10. 2021Return-based Scaling: Yet Another Normalisation Trick for Deep RLcited by 22
  11. 2020Temporal Difference Uncertainties as a Signal for Explorationcited by 15
  12. 2016Adaptive Skills, Adaptive Partitions (ASAP)cited by 11
  13. 2019Adapting Behaviour for Learning Progresscited by 11
  14. 2019Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Studycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 6; 2024: 6; 2025: 2; 2026: 1.

20 papers.

Sort:

By area:

  1. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  2. 2022The Phenomenon of Policy Churncited by 28
  3. 2023Planning Goals for Explorationcited by 28
  4. 2022DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systemscited by 27
  5. 2023Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Groundingcited by 18
  6. 2024Boundless Socratic Learning with Language Gamescited by 12
  7. 2022A Mixture of Surprises for Unsupervised Reinforcement Learningcited by 9
  8. 2022An Evaluation Study of Intrinsic Motivation Techniques applied to Reinforcement Learning over Hard Exploration Environmentscited by 8
  9. 2023Self-Motivated Multi-Agent Explorationcited by 7
  10. 2024Learning World Models for Unconstrained Goal Navigationcited by 6
  11. 2022Collaborative Training of Heterogeneous Reinforcement Learning Agents in Environments with Sparse Rewards: What and When to Share?cited by 4
  12. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learningcited by 3
8 more
  1. 2025β-DQN: Improving Deep Q-Learning By Evolving the Behaviorcited by 3
  2. 2023Scaling Goal-based Exploration via Pruning Proto-goalscited by 2
  3. 2024Decoupling Exploration and Exploitation for Unsupervised Pre-training with Successor Featurescited by 1
  4. 2024Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learningcited by 1
  5. 2023An Autonomous Non-monolithic Agent with Multi-mode Exploration based on Options Frameworkcited by 0
  6. 2024VDSC: Enhancing Exploration Timing with Value Discrepancy and State Countscited by 0
  7. 2025Leveraging Temporally Extended Behavior Sharing for Multi-task Reinforcement Learningcited by 0
  8. 2026SUN: Reaching for Novelty in Reinforcement Learningcited by 0