mlsanityData to 7 Oct 2026

Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation

arXiv 2210.10469 ↗First posted 19 Oct 2022cs.LG cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

A promising paradigm for offline reinforcement learning (RL) is to constrain the learned policy to stay close to the dataset behaviors, known as policy constraint offline RL. However, existing works heavily rely on the purity of the data, exhibiting performance degradation or even catastrophic failure when learning from contaminated datasets containing impure trajectories of diverse levels. e.g., expert level, medium level, etc., while offline contaminated data logs exist commonly in the real world. To mitigate this, we first introduce gradient penalty over the learned value function to tackle the exploding Q-functions. We then relax the closeness constraints towards non-optimal actions with critic weighted constraint relaxation. Experimental results show that the proposed techniques effectively tame the non-optimal trajectories for policy constraint offline RL methods, evaluated on a set of contaminated D4RL Mujoco and Adroit datasets.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 50 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

34 of 50 references.

Closest ancestors · 13

Same direction, newest first

  1. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  2. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  5. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  8. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  9. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  10. 2021Reducing Conservativeness Oriented Offline Reinforcement Learning27 Feb
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
1 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 21

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  6. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  7. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  8. 2019Imitation Learning from Imperfect Demonstrationcited by 112
  9. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  10. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  11. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  12. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44
9 more
  1. 2021Data Poisoning Attacks to Deep Learning Based Recommender Systemscited by 41
  2. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamicscited by 26
  3. 2021Gradient Normalization for Generative Adversarial Networkscited by 25
  4. 2020Lipschitz Lifelong Reinforcement Learningcited by 23
  5. 2019Adversarial Imitation Learning from Incomplete Demonstrationscited by 16
  6. 2021Actor-critic is implicitly biased towards high entropy optimal policiescited by 13
  7. 2019VILD: Variational Imitation Learning with Diverse-quality Demonstrationscited by 12
  8. 2020Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrationscited by 7
  9. 2021On the Benefits of Inducing Local Lipschitzness for Robust Generative Adversarial Imitation Learningcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rolloutcited by 2
  2. 2024HG2P: Hippocampus-inspired High-reward Graph and Model-Free Q-Gradient Penalty for Path Planning and Motion Controlcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
  2. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  3. 2021The Least Restriction for Offline Reinforcement Learning
  4. 2025Online Optimization for Offline Safe Reinforcement Learning
  5. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
  6. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
  7. 2024Offline Safe Reinforcement Learning Using Trajectory Classification
  8. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL