mlsanityData to 14 Oct 2026

ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy

arXiv 2412.03800 ↗First posted 5 Dec 2024cs.LG cs.AIcited by 2

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement learning agents depend on reward signals whose density is rarely under the designer's control, and when such signals are absent, an agent must generate its own drive to explore. State entropy maximization offers a principled objective for this, but existing methods break down at scale in two ways: the intrinsic reward vanishes once a state has been visited, discouraging revisits to the very gateways that lead onward, and estimating entropy over millions of accumulated observations becomes computationally prohibitive. We address both with Episodic and Lifelong Exploration via Maximum Entropy (ELEMENT), a multiscale intrinsically motivated framework for reward-free exploration that transfers to downstream tasks. ELEMENT couples lifelong entropy maximization with a complementary episodic term acting on a faster timescale. For the episodic term, we derive average episodic state entropy, an intrinsic reward that is the exact minimizer of a tractable upper bound on the reward-decomposition objective; for the lifelong term, we propose a kNN graph-based estimator that keeps entropy tractable without forgetting. ELEMENT consistently outperforms state-of-the-art intrinsic reward baselines on state coverage and unsupervised pre-training. Videos, code, and supplementary material: https://sites.google.com/view/element-rl.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 35 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

26 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Curiosity & Entropy Driven Unsupervised RL in Multiple Environments8 Jan

Other references · 25

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  8. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  9. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  10. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  11. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  12. 2021Reinforcement Learning with Prototypical Representationscited by 179
13 more
  1. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  2. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  3. 2019Unsupervised State Representation Learning in Ataricited by 144
  4. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  5. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  6. 2020Intrinsic Reward Driven Imitation Learning via Generative Modelcited by 35
  7. 2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
  8. 2023Fast Rates for Maximum Entropy Explorationcited by 33
  9. 2020Learning Guidance Rewards with Trajectory-space Smoothingcited by 30
  10. 2019Graph-based Nearest Neighbor Search: From Practice to Theorycited by 19
  11. 2022R'enyi State Entropy for Exploration Acceleration in Reinforcement Learningcited by 18
  12. 2022k-Means Maximum Entropy Explorationcited by 11
  13. 2016Fast Online k-nn Graph Buildingcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025State Entropy Regularization for Robust Reinforcement Learningcited by 6
  2. 2025Towards Principled Unsupervised Multi-Agent Reinforcement Learningcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measures
  2. 2024Intrinsic Rewards for Exploration without Harm from Observational Noise: A Simulation Study Based on the Free Energy Principle
  3. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures
  4. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control
  5. 2024Fostering Intrinsic Motivation in Reinforcement Learning with Pretrained Foundation Models
  6. 2022LECO: Learnable Episodic Count for Task-Specific Intrinsic Reward
  7. 2025EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning
  8. 2026Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization