mlsanityData to 14 Oct 2026

Policy Constraint by Only Support Constraint for Offline Reinforcement Learning

arXiv 2503.05207 ↗First posted 7 Mar 2025cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) aims to optimize a policy by using pre-collected datasets, to maximize cumulative rewards. However, offline reinforcement learning suffers challenges due to the distributional shift between the learned and behavior policies, leading to errors when computing Q-values for out-of-distribution (OOD) actions. To mitigate this issue, policy constraint methods aim to constrain the learned policy's distribution with the distribution of the behavior policy or confine action selection within the support of the behavior policy. However, current policy constraint methods tend to exhibit excessive conservatism, hindering the policy from further surpassing the behavior policy's performance. In this work, we present Only Support Constraint (OSC) which is derived from maximizing the total probability of learned policy in the support of behavior policy, to address the conservatism of policy constraint. OSC presents a regularization term that only restricts policies to the support without imposing extra constraints on actions within the support. Additionally, to fully harness the performance of the new policy constraints, OSC utilizes a diffusion model to effectively characterize the support of behavior policies. Experimental evaluations across a variety of offline RL benchmarks demonstrate that OSC significantly enhances performance, alleviating the challenges associated with distributional shifts and mitigating conservatism of policy constraints. Code is available at https://github.com/MoreanP/OSC.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 34 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

30 of 34 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  2. 2023The In-Sample Softmax for Offline Reinforcement Learning28 Feb
  3. 2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning28 Jan
  4. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  5. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  6. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  7. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  10. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  11. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
3 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  3. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 15

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  9. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  10. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  11. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  12. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
3 more
  1. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 149
  2. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  3. 2023Context Shift Reduction for Offline Meta-Reinforcement Learningcited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
  2. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
  3. 2021Behavior Constraining in Weight Space for Offline Reinforcement Learning
  4. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning
  5. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
  6. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation
  7. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation
  8. 2021The Least Restriction for Offline Reinforcement Learning