mlsanityData to 7 Oct 2026

Boosting Exploration in Actor-Critic Algorithms by Incentivizing Plausible Novel States

arXiv 2210.00211 ↗First posted 1 Oct 2022cs.LG cs.SY eess.SYcited by 3

Not on the map. Why?

Abstract (arXiv)

Actor-critic (AC) algorithms are a class of model-free deep reinforcement learning algorithms, which have proven their efficacy in diverse domains, especially in solving continuous control problems. Improvement of exploration (action entropy) and exploitation (expected return) using more efficient samples is a critical issue in AC algorithms. A basic strategy of a learning algorithm is to facilitate indiscriminately exploring all of the environment state space, as well as to encourage exploring rarely visited states rather than frequently visited one. Under this strategy, we propose a new method to boost exploration through an intrinsic reward, based on measurement of a state's novelty and the associated benefit of exploring the state (with regards to policy optimization), altogether called plausible novelty. With incentivized exploration of plausible novel states, an AC algorithm is able to improve its sample efficiency and hence training performance. The new method is verified by extensive simulations of continuous control tasks of MuJoCo environments on a variety of prominent off-policy AC algorithms.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 1 Oct 2022.

Builds on

24 of 38 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2016OpenAI Gymcited by 1,684
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  9. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  10. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  11. 2019Self-Supervised Exploration via Disagreementcited by 263
  12. 2017Noisy Networks for Explorationcited by 262
12 more
  1. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  2. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  3. 2019Model-free Deep Reinforcement Learning for Urban Autonomous Drivingcited by 96
  4. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
  5. 2020Continuous-Discrete Reinforcement Learning for Hybrid Control in Roboticscited by 42
  6. 2018Feedback Control For Cassie With Deep Reinforcement Learningcited by 36
  7. 2019Curiosity-Driven Experience Prioritization via Density Estimationcited by 27
  8. 2018On Passivity, Reinforcement Learning and Higher-Order Learning in Multi-Agent Finite Gamescited by 8
  9. 2021Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experiencecited by 8
  10. 2021Optimal Actor-Critic Policy with Optimized Training Datasetscited by 2
  11. 2022Intrinsically-Motivated Reinforcement Learning: A Brief Introductioncited by 2
  12. 2020Stochastic Optimal Control for Multivariable Dynamical Systems Using Expectation Maximizationcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 2.

3 papers.

  1. 2023A Survey on Physics Informed Reinforcement Learning: Review and Open Problemscited by 22
  2. 2025Off-policy Reinforcement Learning with Model-based Exploration Augmentationcited by 3
  3. 2025Task Specific Sharpness Aware O-RAN Resource Management using Multi Agent Reinforcement Learningcited by 1