mlsanityData to 14 Oct 2026

CROP: Conservative Reward for Model-based Offline Policy Optimization

arXiv 2310.17245 ↗First posted 26 Oct 2023cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) aims to optimize a policy using collected data without online interactions. Model-based approaches are particularly appealing for addressing offline RL challenges because of their capability to mitigate the limitations of data coverage through data generation using models. Nonetheless, a prevalent issue in offline RL is the overestimation caused by distribution shift. This study proposes a novel model-based offline RL algorithm named Conservative Reward for model-based Offline Policy optimization (CROP). CROP introduces a streamlined objective that concurrently minimizes estimation error and the rewards of random actions, thereby yielding a robustly conservative reward estimator. Theoretical analysis shows that the designed conservative reward mechanism leads to a conservative policy evaluation and mitigates distribution shift. Experiments showcase that with the simple modification to reward estimation, CROP can conservatively estimate the reward and achieve competitive performance with existing methods. The source code will be available after acceptance.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 47 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

33 of 47 references.

Closest ancestors · 23

Same direction, newest first

  1. 2023Model-based Offline Reinforcement Learning with Count-based Conservatism21 Jul
  2. 2023Adversarial Model for Offline Reinforcement Learning21 Feb
  3. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief13 Oct
  4. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination16 Jun
  5. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  6. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  7. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  8. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  9. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  10. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  11. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  12. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
11 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  2. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  4. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  5. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  6. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 10

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  4. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  5. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
  6. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  7. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  8. 2016Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networkscited by 49
  9. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 31
  10. 2023CASOG: Conservative Actor-critic with SmOoth Gradient for Skill Learning in Robot-Assisted Interventioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 1; 2026: 1.

3 papers.

  1. 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operatorcited by 9
  2. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentationcited by 2
  3. 2026When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  2. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning
  3. 2024Mitigating Distribution Shift in Model-based Offline RL via Shifts-aware Reward Learning
  4. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation
  5. 2020POPO: Pessimistic Offline Policy Optimization
  6. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  7. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  8. 2023A Survey on Offline Model-Based Reinforcement Learning