mlsanityData to 14 Oct 2026

Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data

arXiv 2507.08761 ↗First posted 11 Jul 2025cs.LG cs.AIcited by 7

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement learning with offline data suffers from Q-value extrapolation errors. To address this issue, we first demonstrate that linear extrapolation of the Q-function beyond the data range is particularly problematic. To mitigate this, we propose guiding the gradual decrease of Q-values outside the data range, which is achieved through reward scaling with layer normalization (RS-LN) and a penalization mechanism for infeasible actions (PA). By combining RS-LN and PA, we develop a new algorithm called PARS. We evaluate PARS across a range of tasks, demonstrating superior performance compared to state-of-the-art algorithms in both offline training and online fine-tuning on the D4RL benchmark, with notable success in the challenging AntMaze Ultra task.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 46 of its 53 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

46 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data10 Dec

Other references · 45

  1. 2016Layer Normalizationcited by 5,669
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  9. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  11. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  12. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
33 more
  1. 2016On the Expressive Power of Deep Neural Networkscited by 306
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  4. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 222
  5. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  6. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  7. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  8. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  9. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  10. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
  11. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  12. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  13. 2022Jump-Start Reinforcement Learningcited by 112
  14. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  15. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  16. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  17. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  18. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 76
  19. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 74
  20. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  21. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  22. 2023Anti-Exploration by Random Network Distillationcited by 50
  23. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  24. 2023Small batch deep reinforcement learningcited by 36
  25. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 26
  26. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuningcited by 20
  27. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
  28. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  29. 2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18
  30. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learningcited by 15
  31. 2023Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agentscited by 12
  32. 2023Goal-Conditioned Predictive Coding for Offline Reinforcement Learningcited by 10
  33. 2024Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  2. 2026FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectificationcited by 2
  3. 2026Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learningcited by 1
  4. 2026Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learningcited by 1
  5. 2026AdamO: A Collapse-Suppressed Optimizer for Offline RLcited by 1
  6. 2026Automatic Constraint Policy Optimization based on Continuous Constraint Interpolation Framework for Offline Reinforcement Learningcited by 0
  7. 2026Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  2. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  3. 2025Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsity
  4. 2024Strategically Conservative Q-Learning
  5. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces
  6. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  7. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
  8. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data