mlsanityData to 7 Oct 2026

Model-Free Active Exploration in Reinforcement Learning

arXiv 2407.00801 ↗First posted 30 Jun 2024cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We study the problem of exploration in Reinforcement Learning and present a novel model-free solution. We adopt an information-theoretical viewpoint and start from the instance-specific lower bound of the number of samples that have to be collected to identify a nearly-optimal policy. Deriving this lower bound along with the optimal exploration strategy entails solving an intricate optimization problem and requires a model of the system. In turn, most existing sample optimal exploration algorithms rely on estimating the model. We derive an approximation of the instance-specific lower bound that only involves quantities that can be inferred using model-free approaches. Leveraging this approximation, we devise an ensemble-based model-free exploration strategy applicable to both tabular and continuous Markov decision processes. Numerical results demonstrate that our strategy is able to identify efficient policies faster than state-of-the-art exploration approaches

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 37 of its 76 references that are in the map. First posted on arXiv 30 Jun 2024.

Builds on

37 of 76 references.

References · 37

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2018Exploration by Random Network Distillationcited by 966
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  6. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 633
  7. 2016Deep Exploration via Bootstrapped DQNcited by 518
  8. 2018Is Q-learning Provably Efficient?cited by 485
  9. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  10. 2017A Tutorial on Thompson Samplingcited by 420
  11. 2014On the Complexity of Best Arm Identification in Multi-Armed Bandit Modelscited by 280
  12. 2017Noisy Networks for Explorationcited by 262
25 more
  1. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  2. 2016Optimal Best Arm Identification with Fixed Confidencecited by 233
  3. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  4. 2017Parameter Space Noise for Explorationcited by 158
  5. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  6. 2017Deep Exploration via Randomized Value Functionscited by 155
  7. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  8. 2019Behaviour Suite for Reinforcement Learningcited by 129
  9. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  10. 2018Information Directed Sampling and Bandits with Heteroscedastic Noisecited by 75
  11. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 68
  12. 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
  13. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 59
  14. 2019Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDPcited by 53
  15. 2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
  16. 2016Learning to Perform Physics Experiments via Deep Reinforcement Learningcited by 47
  17. 2021Beyond No Regret: Instance-Dependent PAC Reinforcement Learningcited by 41
  18. 2015Bootstrapped Thompson Sampling and Deep Explorationcited by 40
  19. 2018Exploration in Structured Reinforcement Learningcited by 34
  20. 2022Instance-Dependent Near-Optimal Policy Identification in Linear MDPs via Online Experiment Designcited by 33
  21. 2021Navigating to the Best Policy in Markov Decision Processescited by 28
  22. 2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
  23. 2020Adaptive Sampling for Best Policy Identification in Markov Decision Processescited by 20
  24. 2022Near Instance-Optimal PAC Reinforcement Learning for Deterministic MDPscited by 17
  25. 2017Efficient exploration with Double Uncertain Value Networkscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2025Embodied AI: From LLMs to World Modelscited by 23
  2. 2025Adaptive Exploration for Multi-Reward Multi-Policy Evaluationcited by 5
  3. 2024Offline Reinforcement Learning and Sequence Modeling for Downlink Link Adaptationcited by 3
  4. 2026DARLING: Detection Augmented Reinforcement Learning with Non-Stationary Guaranteescited by 0
  5. 2026Variance Driven Exploration: A Provable and Efficient Methodology for Pure Exploration in Highly Stochastic Environmentscited by 0