mlsanityData to 7 Oct 2026

Reducing Conservativeness Oriented Offline Reinforcement Learning

arXiv 2103.00098 ↗First posted 27 Feb 2021cs.LGcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In offline reinforcement learning, a policy learns to maximize cumulative rewards with a fixed collection of data. Towards conservative strategy, current methods choose to regularize the behavior policy or learn a lower bound of the value function. However, exorbitant conservation tends to impair the policy's generalization ability and degrade its performance, especially for the mixed datasets. In this paper, we propose the method of reducing conservativeness oriented reinforcement learning. On the one hand, the policy is trained to pay more attention to the minority samples in the static dataset to address the data imbalance problem. On the other hand, we give a tighter lower bound of value function than previous methods to discover potential optimal actions. Consequently, our proposed method is able to tackle the skewed distribution of the provided dataset and derive a value function closer to the expected value function. Experimental results demonstrate that our proposed method outperforms the state-of-the-art methods in D4RL offline reinforcement learning evaluation tasks and our own designed mixed datasets.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 37 references that are in the map. First posted on arXiv 27 Feb 2021.

Builds on

23 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020MOPO: Model-based Offline Policy Optimization27 May
  3. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2018Exploration by Random Network Distillationcited by 966
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  9. 2019Challenges of Real-World Reinforcement Learningcited by 282
  10. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  11. 2016Variational Deep Embedding: An Unsupervised and Generative Approach to Clusteringcited by 219
  12. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 181
6 more
  1. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  2. 2015Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networkscited by 115
  3. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  4. 2018Playing hard exploration games by watching YouTubecited by 98
  5. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  6. 2017End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradientcited by 23

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 0; 2024: 1.

3 papers.

  1. 2022Forgetting and Imbalance in Robot Lifelong Learning with Off-policy Datacited by 5
  2. 2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxationcited by 2
  3. 2024Dataset Clustering for Improved Offline Policy Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  2. 2023Offline Reinforcement Learning with Imbalanced Datasets
  3. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  4. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
  5. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  6. 2024Strategically Conservative Q-Learning
  7. 2023Conservative State Value Estimation for Offline Reinforcement Learning
  8. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning