mlsanityData to 14 Oct 2026

Exploration and Anti-Exploration with Distributional Random Network Distillation

arXiv 2401.09750 ↗First posted 18 Jan 2024cs.LGcited by 33

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Exploration remains a critical issue in deep reinforcement learning for an agent to attain high returns in unknown environments. Although the prevailing exploration Random Network Distillation (RND) algorithm has been demonstrated to be effective in numerous environments, it often needs more discriminative power in bonus allocation. This paper highlights the "bonus inconsistency" issue within RND, pinpointing its primary limitation. To address this issue, we introduce the Distributional RND (DRND), a derivative of the RND. DRND enhances the exploration process by distilling a distribution of random networks and implicitly incorporating pseudo counts to improve the precision of bonus allocation. This refinement encourages agents to engage in more extensive exploration. Our method effectively mitigates the inconsistency issue without introducing significant computational overhead. Both theoretical analysis and experimental results demonstrate the superiority of our approach over the original RND algorithm. Our method excels in challenging online exploration scenarios and effectively serves as an anti-exploration mechanism in D4RL offline tasks. Our code is publicly available at https://github.com/yk7333/DRND.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 40 of its 49 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

40 of 49 references.

References · 40

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  9. 2018Exploration by Random Network Distillationcited by 993
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  12. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
28 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  2. 2016Deep Exploration via Bootstrapped DQNcited by 527
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  4. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  5. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  6. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  7. 2020Critic Regularized Regressioncited by 275
  8. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  9. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  10. 2017Count-Based Exploration with Neural Density Modelscited by 257
  11. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
  12. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  13. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  14. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
  15. 2018Count-Based Exploration with the Successor Representationcited by 112
  16. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  17. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 84
  18. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  19. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 76
  20. 2023Anti-Exploration by Random Network Distillationcited by 50
  21. 2017Count-Based Exploration in Feature Space for Reinforcement Learningcited by 49
  22. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  23. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  24. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
  25. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  26. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learningcited by 20
  27. 2023Model-based Offline Reinforcement Learning with Count-based Conservatismcited by 16
  28. 2022State Advantage Weighting for Offline RLcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 15; 2026: 8.

33 papers.

Sort:

By area:

  1. 2026Maximum Likelihood Reinforcement Learningcited by 36
  2. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  3. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 22
  4. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  5. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Modelscited by 14
  6. 2025EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Controlcited by 13
  7. 2024World Models with Hints of Large Language Models for Goal Achievingcited by 6
  8. 2025FastFT: Accelerating Reinforced Feature Transformation via Advanced Exploration Strategiescited by 5
  9. 2024Deep Exploration with PAC-Bayescited by 4
  10. 2024Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learningcited by 4
  11. 2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
  12. 2024The Role of Deep Learning Regularizations on Actors in Offline RLcited by 2
21 more
  1. 2025Overcoming Non-stationary Dynamics with Evidential Proximal Policy Optimizationcited by 2
  2. 2025Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoringcited by 2
  3. 2025Finite-time Convergence Analysis of Actor-Critic with Evolving Rewardcited by 2
  4. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  5. 2024Novelty-based Sample Reuse for Continuous Robotics Controlcited by 1
  6. 2025Exploration by Random Distribution Distillationcited by 1
  7. 2025Exploration by Random Reward Perturbationcited by 1
  8. 2025Offline Reinforcement Learning with Penalized Action Noise Injectioncited by 1
  9. 2025From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learningcited by 1
  10. 2025Mixture of Autoencoder Experts Guidance using Unlabeled and Incomplete Data for Exploration in Reinforcement Learningcited by 1
  11. 2025Learning Without Time-Based Embodiment Resets in Soft-Actor Criticcited by 1
  12. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correctioncited by 1
  13. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  14. 2024CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learningcited by 0
  15. 2025Coupled Distributional Random Expert Distillation for World Model Online Imitation Learningcited by 0
  16. 2025ObjectRL: An Object-Oriented Reinforcement Learning Codebasecited by 0
  17. 2025CDE: Concept-Driven Exploration for Reinforcement Learningcited by 0
  18. 2026Distributional Active Inferencecited by 0
  19. 2026Intrinsic Reward Policy Optimization for Sparse-Reward Environmentscited by 0
  20. 2026CIG: Exploration via Conditional Information Gaincited by 0
  21. 2026Information-Based Exploration via Random Features for Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Random Latent Exploration for Deep Reinforcement Learning
  2. 2024VDSC: Enhancing Exploration Timing with Value Discrepancy and State Counts
  3. 2025Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximization
  4. 2024Random Network Distillation Based Deep Reinforcement Learning for AGV Path Planning
  5. 2025Value of Information-Enhanced Exploration in Bootstrapped DQN
  6. 2026Decoupling Exploration from Optimization in RLVR
  7. 2024Efficient Exploration in Deep Reinforcement Learning: A Novel Bayesian Actor-Critic Algorithm
  8. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization