mlsanityData to 14 Oct 2026

Exploration by Running Away from the Past

arXiv 2411.14085 ↗First posted 21 Nov 2024cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

The ability to explore efficiently and effectively is a central challenge of reinforcement learning. In this work, we consider exploration through the lens of information theory. Specifically, we cast exploration as a problem of maximizing the Shannon entropy of the state occupation measure. This is done by maximizing a sequence of divergences between distributions representing an agent's past behavior and its current behavior. Intuitively, this encourages the agent to explore new behaviors that are distinct from past behaviors. Hence, we call our method RAMP, for ``Running Away from the Past.'' A fundamental question of this method is the quantification of the distribution change over time. We consider both the Kullback-Leibler divergence and the Wasserstein distance to quantify divergence between successive state occupation measures, and explain why the former might lead to undesirable exploratory behaviors in some tasks. We demonstrate that by encouraging the agent to explore by actively distancing itself from past experiences, it can effectively explore mazes and a wide range of behaviors on robotic manipulation and locomotion tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 16 of its 39 references that are in the map. First posted on arXiv 21 Nov 2024.

Builds on

16 of 39 references.

References · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2018Exploration by Random Network Distillationcited by 993
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  5. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  6. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  7. 2016Variational Intrinsic Controlcited by 267
  8. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  9. 2019Efficient Exploration via State Marginal Matchingcited by 188
  10. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 103
  11. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  12. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
4 more
  1. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 68
  2. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  3. 2021Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reachingcited by 19
  4. 2022Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environmentscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy
  2. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measures
  3. 2024Intrinsic Rewards for Exploration without Harm from Observational Noise: A Simulation Study Based on the Free Energy Principle
  4. 2026Explore Beyond the Boundary Using Entropic Information
  5. 2024The Exploration-Exploitation Dilemma Revisited: An Entropy Perspective
  6. 2024Behind the Myth of Exploration in Policy Gradients
  7. 2025Exploration Behavior of Untrained Policies
  8. 2024Deterministic Exploration via Stationary Bellman Error Maximization