mlsanityData to 7 Oct 2026

Model-based Offline Reinforcement Learning with Count-based Conservatism

arXiv 2307.11352 ↗First posted 21 Jul 2023cs.LG stat.MLcited by 16

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In this paper, we propose a model-based offline reinforcement learning method that integrates count-based conservatism, named textttCount-MORL. Our method utilizes the count estimates of state-action pairs to quantify model estimation error, marking the first algorithm of demonstrating the efficacy of count-based conservatism in model-based offline deep RL to the best of our knowledge. For our proposed method, we first show that the estimation error is inversely proportional to the frequency of state-action pairs. Secondly, we demonstrate that the learned policy under the count-based conservative model offers near-optimality performance guarantees. Through extensive numerical experiments, we validate that textttCount-MORL with hash code implementation significantly outperforms existing offline RL algorithms on the D4RL benchmark datasets. The code is accessible at hrefhttps://github.com/oh-lab/Count-MORLhttps://github.com/oh-lab/Count-MORL.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 53 references that are in the map. First posted on arXiv 21 Jul 2023.

Builds on

39 of 53 references.

Closest ancestors · 19

Same direction, newest first

  1. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning8 Dec
  2. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  3. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  8. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  12. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
7 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020MOPO: Model-based Offline Policy Optimization27 May
  3. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 20

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  6. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  7. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  8. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  9. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  10. 2017Count-Based Exploration with Neural Density Modelscited by 255
  11. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  12. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
8 more
  1. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
  2. 2020Model-Based Offline Planningcited by 126
  3. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  4. 2018Count-Based Exploration with the Successor Representationcited by 112
  5. 2017Count-Based Exploration in Feature Space for Reinforcement Learningcited by 48
  6. 2017EX2: Exploration with Exemplar Models for Deep Reinforcement Learningcited by 42
  7. 2021TempoRL: Learning When to Actcited by 25
  8. 2022Model-Based Reinforcement Learning with Multinomial Logistic Function Approximationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 1; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
  2. 2024SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learningcited by 12
  3. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Sufficescited by 11
  4. 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operatorcited by 9
  5. 2023CROP: Conservative Reward for Model-based Offline Policy Optimizationcited by 3
  6. 2024Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learningcited by 2
  7. 2024Decision-Point Guided Safe Policy Improvementcited by 2
  8. 2026Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learningcited by 2
  9. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weightingcited by 2
  10. 2024Differentially Private Deep Model-Based Reinforcement Learningcited by 1
  11. 2024Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learningcited by 1
  12. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
4 more
  1. 2024Hypercube Policy Regularization Framework for Offline Reinforcement Learningcited by 0
  2. 2024Bayesian Inverse Transition Learning: Learning Dynamics From Near-Optimal Trajectoriescited by 0
  3. 2026Efficient Anti-exploration via VQVAE and Fuzzy Clustering in Offline Reinforcement Learningcited by 0
  4. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
  2. 2021Offline Reinforcement Learning with Reverse Model-based Imagination
  3. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning
  4. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL
  5. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  6. 2025Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modeling
  7. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  8. 2021Offline RL Without Off-Policy Evaluation