mlsanityData to 14 Oct 2026

The Exploration-Exploitation Dilemma Revisited: An Entropy Perspective

arXiv 2408.09974 ↗First posted 19 Aug 2024cs.LGcited by 5

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

The imbalance of exploration and exploitation has long been a significant challenge in reinforcement learning. In policy optimization, excessive reliance on exploration reduces learning efficiency, while over-dependence on exploitation might trap agents in local optima. This paper revisits the exploration-exploitation dilemma from the perspective of entropy by revealing the relationship between entropy and the dynamic adaptive process of exploration and exploitation. Based on this theoretical insight, we establish an end-to-end adaptive framework called AdaZero, which automatically determines whether to explore or to exploit as well as their balance of strength. Experiments show that AdaZero significantly outperforms baseline models across various Atari and MuJoCo environments with only a single setting. Especially in the challenging environment of Montezuma, AdaZero boosts the final returns by up to fifteen times. Moreover, we conduct a series of visualization analyses to reveal the dynamics of our self-adaptive mechanism, demonstrating how entropy reflects and changes with respect to the agent's performance and adaptive process.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 34 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

18 of 34 references.

References · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2018Exploration by Random Network Distillationcited by 993
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  8. 2017Count-Based Exploration with Neural Density Modelscited by 257
  9. 2018Self-Imitation Learningcited by 138
  10. 2020Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrificescited by 59
  11. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
  12. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
6 more
  1. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learningcited by 31
  2. 2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
  3. 2020Latent World Models For Intrinsically Motivated Explorationcited by 15
  4. 2021Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learningcited by 14
  5. 2023LESSON: Learning to Integrate Exploration Strategies for Reinforcement Learning via an Option Frameworkcited by 7
  6. 2021Entropy Regularized Reinforcement Learning Using Large Deviation Theorycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?cited by 6
  2. 2026Pixel-Space Diffusion Transformerscited by 1
  3. 2026Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Modelscited by 1
  4. 2025Cavity Duplexer Tuning with 1d Resnet-like Neural Networkscited by 0
  5. 2026PAST: Prompt-Adaptive Sampling Termination for Efficient Diffusion Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning
  2. 2025A Novel Framework for Uncertainty-Driven Adaptive Exploration
  3. 2024ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy
  4. 2025Exploitation Is All You Need... for Exploration
  5. 2025Evolutionary Policy Optimization
  6. 2024Behind the Myth of Exploration in Policy Gradients
  7. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control
  8. 2026Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration