mlsanityData to 14 Oct 2026

Exclusively Penalized Q-learning for Offline Reinforcement Learning

arXiv 2405.14082 ↗First posted 23 May 2024cs.LG cs.AIcited by 8

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Constraint-based offline reinforcement learning (RL) involves policy constraints or imposing penalties on the value function to mitigate overestimation errors caused by distributional shift. This paper focuses on a limitation in existing offline RL methods with penalized value function, indicating the potential for underestimation bias due to unnecessary bias introduced in the value function. To address this concern, we propose Exclusively Penalized Q-learning (EPQ), which reduces estimation bias in the value function by selectively penalizing states that are prone to inducing estimation errors. Numerical results show that our method significantly reduces underestimation bias and improves performance in various offline control tasks compared to other offline RL methods

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 57 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

49 of 57 references.

Closest ancestors · 23

Same direction, newest first

  1. 2023The In-Sample Softmax for Offline Reinforcement Learning28 Feb
  2. 2023Behavior Proximal Policy Optimization22 Feb
  3. 2022Mutual Information Regularized Offline Reinforcement Learning14 Oct
  4. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  5. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  6. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  7. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  8. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  9. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning31 Jan
  10. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  11. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  12. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
11 more
  1. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  2. 2021Offline RL Without Off-Policy Evaluation16 Jun
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  5. 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  7. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  8. 2016Generative Adversarial Imitation Learningcited by 1,597
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  11. 2017Hindsight Experience Replaycited by 1,136
  12. 2015Prioritized Experience Replaycited by 1,099
14 more
  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  3. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  4. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  5. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  6. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 86
  7. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  8. 2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
  9. 2022Robust Imitation Learning against Variations in Environment Dynamicscited by 26
  10. 2020Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Explorationcited by 20
  11. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 17
  12. 2019Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient Reinforcement Learningcited by 12
  13. 2023Domain Adaptive Imitation Learning with Visual Observationcited by 11
  14. 2023FoX: Formation-aware exploration in multi-agent reinforcement learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  2. 2025Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Taskscited by 3
  3. 2026Robust Regularized Policy Iteration under Transition Uncertaintycited by 2
  4. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correctioncited by 1
  5. 2026Offline Policy Optimization with Posterior Samplingcited by 1
  6. 2025CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimizationcited by 0
  7. 2026Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learningcited by 0
  8. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  3. 2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation
  4. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  5. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model
  6. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  7. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning
  8. 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage