# BeBold: Exploration Beyond the Boundary of Explored Regions

arXiv: https://arxiv.org/abs/2012.08621 · HTML: https://mlsanity.com/paper/2012.08621/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2020.

## Built on by (most cited first, 30 of 33)

- [MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research](https://mlsanity.com/paper/2109.13202/) (2021)
- [XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX](https://mlsanity.com/paper/2312.12044/) (2023)
- [MADE: Exploration via Maximizing Deviation from Explored Regions](https://mlsanity.com/paper/2106.10268/) (2021)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [The SocialAI School: Insights from Developmental Psychology Towards Artificial Socio-Cultural Agents](https://mlsanity.com/paper/2307.07871/) (2023)
- [Dynamic Bottleneck for Robust Self-Supervised Exploration](https://mlsanity.com/paper/2110.10735/) (2021)
- [TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Play](https://mlsanity.com/paper/2302.07515/) (2023)
- [TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations](https://mlsanity.com/paper/2110.04507/) (2021)
- [NAVIX: Scaling MiniGrid Environments with JAX](https://mlsanity.com/paper/2407.19396/) (2024)
- [C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks](https://mlsanity.com/paper/2110.12080/) (2021)
- [Generalization in Text-based Games via Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2109.09968/) (2021)
- [minimax: Efficient Baselines for Autocurricula in JAX](https://mlsanity.com/paper/2311.12716/) (2023)
- [RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learning](https://mlsanity.com/paper/2405.19548/) (2024)
- [Focus on Impact: Indoor Exploration with Intrinsic Motivation](https://mlsanity.com/paper/2109.08521/) (2021)
- [Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness](https://mlsanity.com/paper/2105.09992/) (2021)
- [Generative Personas That Behave and Experience Like Humans](https://mlsanity.com/paper/2209.00459/) (2022)
- [Explore and Control with Adversarial Surprise](https://mlsanity.com/paper/2107.07394/) (2021)
- [Play with Emotion: Affect-Driven Reinforcement Learning](https://mlsanity.com/paper/2208.12622/) (2022)
- [Approximate information state based convergence analysis of recurrent Q-learning](https://mlsanity.com/paper/2306.05991/) (2023)
- [Maze Learning using a Hyperdimensional Predictive Processing Cognitive Architecture](https://mlsanity.com/paper/2204.00619/) (2022)
- [Learning to Solve Tasks with Exploring Prior Behaviours](https://mlsanity.com/paper/2307.02889/) (2023)
- [β-DQN: Improving Deep Q-Learning By Evolving the Behavior](https://mlsanity.com/paper/2501.00913/) (2025)
- [AI Finds A Way](https://mlsanity.com/paper/2608.23875/) (2026)
- [Co-Imitation Learning without Expert Demonstration](https://mlsanity.com/paper/2103.14823/) (2021)
- [BIMRL: Brain Inspired Meta Reinforcement Learning](https://mlsanity.com/paper/2210.16530/) (2022)
- [A Neuro-mimetic Realization of the Common Model of Cognition via Hebbian Learning and Free Energy Minimization](https://mlsanity.com/paper/2310.15177/) (2023)
- [Learning Curricula in Open-Ended Worlds](https://mlsanity.com/paper/2312.03126/) (2023)
- [OpenRL: A Unified Reinforcement Learning Framework](https://mlsanity.com/paper/2312.16189/) (2023)
- [Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments](https://mlsanity.com/paper/2304.09825/) (2023)
- [Is Exploration or Optimization the Problem for Deep Reinforcement Learning?](https://mlsanity.com/paper/2508.01329/) (2025)

## Abstract (arXiv)

Efficient exploration under sparse rewards remains a key challenge in deep reinforcement learning. To guide exploration, previous work makes extensive use of intrinsic reward (IR). There are many heuristics for IR, including visitation counts, curiosity, and state-difference. In this paper, we analyze the pros and cons of each method and propose the regulated difference of inverse visitation counts as a simple but effective criterion for IR. The criterion helps the agent explore Beyond the Boundary of explored regions and mitigates common issues in count-based methods, such as short-sightedness and detachment. The resulting method, BeBold, solves the 12 most challenging procedurally-generated tasks in MiniGrid with just 120M environment steps, without any curriculum learning. In comparison, the previous SoTA only solves 50% of the tasks. BeBold also achieves SoTA on multiple tasks in NetHack, a popular rogue-like game that contains more challenging procedurally-generated environments.