mlsanityData to 14 Oct 2026

CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learning

arXiv 2503.18980 ↗First posted 23 Mar 2025stat.ML cs.AI cs.LGcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Exploration remains a fundamental challenge in reinforcement learning, as many existing methods either lack theoretical guarantees or fall short in practical effectiveness. In this paper, we propose CAE, i.e., the Critic as an Explorer, a lightweight approach that repurposes the value networks in standard deep RL algorithms to drive exploration, without introducing additional parameters. CAE leverages multi-armed bandit techniques combined with a tailored scaling strategy, enabling efficient exploration with provable sub-linear regret bounds and strong empirical stability. Remarkably, it is simple to implement, requiring only about 10 lines of code. For complex tasks where learning reliable value networks is difficult, we introduce CAE+, an extension of CAE that incorporates an auxiliary network. CAE+ increases the parameter count by less than 1% while preserving implementation simplicity, adding roughly 10 additional lines of code. Extensive experiments on MuJoCo, MiniHack, and Habitat validate the effectiveness of CAE and CAE+, highlighting their ability to unify theoretical rigor with practical efficiency.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 46 of its 69 references that are in the map. First posted on arXiv 23 Mar 2025.

Builds on

46 of 69 references.

References · 46

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  9. 2018Exploration by Random Network Distillationcited by 993
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  11. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  12. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
34 more
  1. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  2. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  3. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  4. 2018Is Q-learning Provably Efficient?cited by 487
  5. 2017On Kernelized Multi-armed Banditscited by 352
  6. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  7. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  8. 2019Provably Efficient Exploration in Policy Optimizationcited by 205
  9. 2020The NetHack Learning Environmentcited by 198
  10. 2013Finite-Time Analysis of Kernelised Contextual Banditscited by 177
  11. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 169
  12. 2017Deep Exploration via Randomized Value Functionscited by 156
  13. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  14. 2020Neural Thompson Samplingcited by 128
  15. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  16. 2019Frequentist Regret Bounds for Randomized Least-Squares Value Iterationcited by 101
  17. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
  18. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 92
  19. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 85
  20. 2020Neural Contextual Bandits with Deep Representation and Shallow Explorationcited by 71
  21. 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
  22. 2022Exploration via Elliptical Episodic Bonusescited by 51
  23. 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
  24. 2021Adversarially Guided Actor-Criticcited by 47
  25. 2014Efficient Learning in Large-Scale Combinatorial Semi-Banditscited by 36
  26. 2023Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlocited by 29
  27. 2022From Dirichlet to Rubin: Optimistic Exploration in RL without Bonusescited by 22
  28. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 20
  29. 2019Deep Neural Linear Bandits: Overcoming Catastrophic Forgetting through Likelihood Matchingcited by 17
  30. 2023Automatic Intrinsic Reward Shaping for Exploration in Deep Reinforcement Learningcited by 12
  31. 2024More Efficient Randomized Exploration for Reinforcement Learning via Approximate Samplingcited by 12
  32. 2021Anti-Concentrated Confidence Bonuses for Scalable Explorationcited by 7
  33. 2023Model-free Posterior Sampling via Learning Rate Randomizationcited by 6
  34. 2024A Contextual Combinatorial Bandit Approach to Negotiationcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Exploration in Deep Reinforcement Learning: A Novel Bayesian Actor-Critic Algorithm
  2. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
  3. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization
  4. 2026Decoupling Exploration from Optimization in RLVR
  5. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  6. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning
  7. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying
  8. 2025BroRL: Scaling Reinforcement Learning via Broadened Exploration