mlsanityData to 7 Oct 2026

Anti-Exploration by Random Network Distillation

arXiv 2301.13616 ↗First posted 31 Jan 2023cs.LG cs.AI cs.NEcited by 49

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Despite the success of Random Network Distillation (RND) in various domains, it was shown as not discriminative enough to be used as an uncertainty estimator for penalizing out-of-distribution actions in offline reinforcement learning. In this paper, we revisit these results and show that, with a naive choice of conditioning for the RND prior, it becomes infeasible for the actor to effectively minimize the anti-exploration bonus and discriminativity is not an issue. We show that this limitation can be avoided with conditioning based on Feature-wise Linear Modulation (FiLM), resulting in a simple and efficient ensemble-free algorithm based on Soft Actor-Critic. We evaluate it on the D4RL benchmark, showing that it is capable of achieving performance comparable to ensemble-based methods and outperforming ensemble-free approaches by a wide margin.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 47 of its 55 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

47 of 55 references.

Closest ancestors · 20

Same direction, newest first

  1. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning8 Dec
  2. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes28 Nov
  3. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size20 Nov
  4. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  5. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  6. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  7. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  8. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  9. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  10. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters8 Oct
  11. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  12. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
8 more
  1. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  2. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016Layer Normalizationcited by 5,579
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  7. 2015Continuous control with deep reinforcement learningcited by 3,279
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  10. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  11. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
15 more
  1. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  2. 2018Exploration by Random Network Distillationcited by 966
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  4. 2022A Generalist Agentcited by 802
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  6. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  7. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  8. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  9. 2022Multi-Game Decision Transformerscited by 212
  10. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  11. 2020Masksembles for Uncertainty Estimationcited by 88
  12. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  13. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 68
  14. 2022Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flowscited by 21
  15. 2022Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environmentscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 13; 2025: 17; 2026: 12.

49 papers.

Sort:

By area:

  1. 2025Flow Q-Learningcited by 162
  2. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  3. 2025Horizon Reduction Makes RL Scalablecited by 55
  4. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  5. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  6. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
  7. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  8. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  9. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 16
  10. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learningcited by 12
  11. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  12. 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
37 more
  1. 2026Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generationcited by 9
  2. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  3. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
  4. 2023Building Open-Ended Embodied Agent via Language-Policy Bidirectional Adaptationcited by 6
  5. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replaycited by 6
  6. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  7. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  8. 2023Katakomba: Tools and Benchmarks for Data-Driven NetHackcited by 5
  9. 2025FastFT: Accelerating Reinforced Feature Transformation via Advanced Exploration Strategiescited by 5
  10. 2024Deep Exploration with PAC-Bayescited by 4
  11. 2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?cited by 4
  12. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learningcited by 4
  13. 2024Distilling LLMs' Decomposition Abilities into Compact Language Modelscited by 3
  14. 2024Offline Reinforcement Learning with Behavioral Supervisor Tuningcited by 3
  15. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learningcited by 3
  16. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spacescited by 3
  17. 2024The Role of Deep Learning Regularizations on Actors in Offline RLcited by 2
  18. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filteringcited by 2
  19. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 2
  20. 2024Strategically Conservative Q-Learningcited by 1
  21. 2024Offline Model-Based Reinforcement Learning with Anti-Explorationcited by 1
  22. 2025Offline Reinforcement Learning with Penalized Action Noise Injectioncited by 1
  23. 2025From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learningcited by 1
  24. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learningcited by 1
  25. 2026Fisher Decorator: Refining Flow Policy via a Local Transport Mapcited by 1
  26. 2026Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learningcited by 1
  27. 2025SALE-Based Offline Reinforcement Learning with Ensemble Q-Networkscited by 0
  28. 2025AMPED: Adaptive Multi-objective Projection for balancing Exploration and skill Diversificationcited by 0
  29. 2025Noise-Guided Transport for Imitation Learningcited by 0
  30. 2025ASTRO: Adaptive Stitching via Dynamics-Guided Trajectory Rolloutscited by 0
  31. 2026Beyond Static Datasets: Robust Offline Policy Optimization via Vetted Synthetic Transitionscited by 0
  32. 2026Efficient Anti-exploration via VQVAE and Fuzzy Clustering in Offline Reinforcement Learningcited by 0
  33. 2026On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inferencecited by 0
  34. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learningcited by 0
  35. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  36. 2026Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learningcited by 0
  37. 2026Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating Diffusion Planners in Offline RL via Reward-Aware Consistency Trajectory Distillation
  2. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  3. 2024Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning
  4. 2025Statistical Guarantees for Offline Domain Randomization
  5. 2024Offline Behavior Distillation
  6. 2026Epistemic Robust Offline Reinforcement Learning
  7. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approach
  8. 2021The Least Restriction for Offline Reinforcement Learning