mlsanityData to 14 Oct 2026

Offline Reinforcement Learning as Anti-Exploration

arXiv 2106.06431 ↗First posted 11 Jun 2021cs.LGcited by 44

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning (RL) aims at learning an optimal control from a fixed dataset, without interactions with the system. An agent in this setting should avoid selecting actions whose consequences cannot be predicted from the data. This is the converse of exploration in RL, which favors such actions. We thus take inspiration from the literature on bonus-based exploration to design a new offline RL agent. The core idea is to subtract a prediction-based exploration bonus from the reward, instead of adding it for exploration. This allows the policy to stay close to the support of the dataset. We connect this approach to a more common regularization of the learned policy towards the data. Instantiated with a bonus based on the prediction error of a variational autoencoder, we show that our agent is competitive with the state of the art on a set of continuous control locomotion and manipulation tasks.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 41 of its 64 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

41 of 64 references.

Closest ancestors · 12

Same direction, newest first

  1. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  2. 2021Regularized Behavior Value Estimation17 Mar
  3. 2021Offline Reinforcement Learning with Pseudometric Learning2 Mar
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  5. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  6. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  7. 2020Critic Regularized Regression26 Jun
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 29

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  7. 2018Exploration by Random Network Distillationcited by 993
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  10. 2017Deep Reinforcement Learning that Matterscited by 754
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
17 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  2. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 403
  3. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  4. 2019A Theory of Regularized Markov Decision Processescited by 286
  5. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  6. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 225
  7. 2021Reinforcement learning based recommender systems: A surveycited by 188
  8. 2018Latent Space Autoregression for Novelty Detectioncited by 149
  9. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  10. 2018Generative Probabilistic Novelty Detection with Adversarial Autoencoderscited by 99
  11. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  12. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  13. 2015Learning Deep Representations of Appearance and Motion for Anomalous Event Detectioncited by 75
  14. 2020Munchausen Reinforcement Learningcited by 69
  15. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 53
  16. 2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
  17. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimationcited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 14; 2024: 8; 2025: 9; 2026: 3.

44 papers.

Sort:

By area:

  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  2. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  3. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  4. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  5. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  6. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 85
  7. 2023Anti-Exploration by Random Network Distillationcited by 50
  8. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  9. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 36
  10. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 33
  11. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  12. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
32 more
  1. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21
  2. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 21
  3. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learningcited by 20
  4. 2023Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RLcited by 15
  5. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraintscited by 13
  6. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
  7. 2023Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learningcited by 13
  8. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  9. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  10. 2022Optimizing Pessimism in Dynamic Treatment Regimes: A Bayesian Learning Approachcited by 7
  11. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learningcited by 7
  12. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularizationcited by 3
  13. 2024Offline Reinforcement Learning with Behavioral Supervisor Tuningcited by 3
  14. 2025Provable Zero-Shot Generalization in Offline Reinforcement Learningcited by 3
  15. 2026Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learningcited by 3
  16. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 3
  17. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flowscited by 2
  18. 2025Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learningcited by 2
  19. 2025Scalable LinUCB: Low-Rank Design Matrix Updates for Recommenders with Large Action Spacescited by 2
  20. 2024Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learningcited by 1
  21. 2024Sparsity-based Safety Conservatism for Constrained Offline Reinforcement Learningcited by 1
  22. 2024Offline Model-Based Reinforcement Learning with Anti-Explorationcited by 1
  23. 2025Offline Diffusion Policy for Multi-User Delay-Constrained Schedulingcited by 1
  24. 2025Beyond Shallow Behavior: Task-Efficient Value-Based Multi-Task Offline MARL via Skill Discoverycited by 1
  25. 2025Offline Reinforcement Learning with Penalized Action Noise Injectioncited by 1
  26. 2022Age of Semantics in Cooperative Communications: To Expedite Simulation Towards Real via Offline Reinforcement Learningcited by 0
  27. 2023Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Explorationcited by 0
  28. 2024Hypercube Policy Regularization Framework for Offline Reinforcement Learningcited by 0
  29. 2025Decision Flow Policy Optimizationcited by 0
  30. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learningcited by 0
  31. 2025Noise-Guided Transport for Imitation Learningcited by 0
  32. 2026Efficient Anti-exploration via VQVAE and Fuzzy Clustering in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022The Challenges of Exploration for Offline Reinforcement Learning
  2. 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationarity
  3. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
  4. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  5. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
  6. 2021A Minimalist Approach to Offline Reinforcement Learning
  7. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
  8. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data