mlsanityData to 14 Oct 2026

OVD-Explorer: Optimism Should Not Be the Sole Pursuit of Exploration in Noisy Environments

arXiv 2312.12145 ↗First posted 19 Dec 2023cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning, the optimism in the face of uncertainty (OFU) is a mainstream principle for directing exploration towards less explored areas, characterized by higher uncertainty. However, in the presence of environmental stochasticity (noise), purely optimistic exploration may lead to excessive probing of high-noise areas, consequently impeding exploration efficiency. Hence, in exploring noisy environments, while optimism-driven exploration serves as a foundation, prudent attention to alleviating unnecessary over-exploration in high-noise areas becomes beneficial. In this work, we propose Optimistic Value Distribution Explorer (OVD-Explorer) to achieve a noise-aware optimistic exploration for continuous control. OVD-Explorer proposes a new measurement of the policy's exploration ability considering noise in optimistic perspectives, and leverages gradient ascent to drive exploration. Practically, OVD-Explorer can be easily integrated with continuous control RL algorithms. Extensive evaluations on the MuJoCo and GridChaos tasks demonstrate the superiority of OVD-Explorer in achieving noise-aware optimistic exploration.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 34 of its 48 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

34 of 48 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  7. 2018MINE: Mutual Information Neural Estimationcited by 683
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  9. 2016Deep Exploration via Bootstrapped DQNcited by 527
  10. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  11. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  12. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
22 more
  1. 2019Self-Supervised Exploration via Disagreementcited by 265
  2. 2018Safe Exploration in Continuous Action Spacescited by 247
  3. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 235
  4. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  5. 2017Max-value Entropy Search for Efficient Bayesian Optimizationcited by 206
  6. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  7. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  8. 2018Information Directed Sampling and Bandits with Heteroscedastic Noisecited by 76
  9. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 70
  10. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 68
  11. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  12. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 60
  13. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
  14. 2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
  15. 2017Count-Based Exploration in Feature Space for Reinforcement Learningcited by 49
  16. 2020Multi-Fidelity Bayesian Optimization via Deep Neural Networkscited by 46
  17. 2021Which Mutual-Information Representation Learning Objectives are Sufficient for Control?cited by 29
  18. 2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
  19. 2022Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learningcited by 26
  20. 2019Constrained Bayesian Optimization with Max-Value Entropy Searchcited by 20
  21. 2020Improved Max-value Entropy Search for Multi-objective Bayesian Optimization with Constraintscited by 15
  22. 2022EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Modelcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 1.

7 papers.

  1. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  2. 2026SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewardscited by 9
  3. 2024Deep Exploration with PAC-Bayescited by 4
  4. 2025Optimistic Exploration for Risk-Averse Constrained Reinforcement Learningcited by 3
  5. 2024vMFER: Von Mises-Fisher Experience Resampling Based on Uncertainty of Gradient Directions for Policy Improvementcited by 1
  6. 2025LNUCB-TA: Linear-nonlinear Hybrid Bandit Learning with Temporal Attentioncited by 1
  7. 2025Multi-Actor Multi-Critic Deep Deterministic Reinforcement Learning with a Novel Q-Ensemble Methodcited by 0