# Neighboring State-based Exploration for Reinforcement Learning

arXiv: https://arxiv.org/abs/2212.10712 · HTML: https://mlsanity.com/paper/2212.10712/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 10 references that are in the map.
- First posted on arXiv 21 Dec 2022.

## Builds on (most cited first, 8 of 8)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Provably Efficient Maximum Entropy Exploration](https://mlsanity.com/paper/1812.02690/) (2018)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [Evading Adversarial Example Detection Defenses with Orthogonal Projected Gradient Descent](https://mlsanity.com/paper/2106.15023/) (2021)

## Abstract (arXiv)

Reinforcement Learning is a powerful tool to model decision-making processes. However, it relies on an exploration-exploitation trade-off that remains an open challenge for many tasks. In this work, we study neighboring state-based, model-free exploration led by the intuition that, for an early-stage agent, considering actions derived from a bounded region of nearby states may lead to better actions when exploring. We propose two algorithms that choose exploratory actions based on a survey of nearby states, and find that one of our methods, ρ-explore, consistently outperforms the Double DQN baseline in an discrete environment by 49% in terms of Eval Reward Return.