mlsanityData to 7 Oct 2026

Automatic Constraint Policy Optimization based on Continuous Constraint Interpolation Framework for Offline Reinforcement Learning

arXiv 2601.23010 ↗First posted 30 Jan 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning (RL) relies on policy constraints to mitigate extrapolation error, where both the constraint form and constraint strength critically shape performance. However, most existing methods commit to a single constraint family: weighted behavior cloning, density regularization, or support constraints, without a unified principle that explains their connections or trade-offs. In this work, we propose Continuous Constraint Interpolation (CCI), a unified optimization framework in which these three constraint families arise as special cases along a common constraint spectrum. The CCI framework introduces a single interpolation parameter that enables smooth transitions and principled combinations across constraint types. Building on CCI, we develop Automatic Constraint Policy Optimization (ACPO), a practical primal--dual algorithm that adapts the interpolation parameter via a Lagrangian dual update. Moreover, we establish a maximum-entropy performance difference lemma and derive performance lower bounds for both the closed-form optimal policy and its parametric projection. Experiments on D4RL and NeoRL2 demonstrate robust gains across diverse domains, achieving state-of-the-art performance overall.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 50 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

36 of 50 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Offline Reinforcement Learning with Wasserstein Regularization via Optimal Transport Maps14 Jul
  2. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL26 May
  3. 2025PIQL: Projective Implicit Q-Learning with Support Constraint for Offline Reinforcement Learning15 Jan
  4. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning22 Dec
  5. 2024Rethinking Optimal Transport in Offline Reinforcement Learning17 Oct
  6. 2023Supported Trust Region Optimization for Offline Reinforcement Learning15 Nov
  7. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  8. 2023The In-Sample Softmax for Offline Reinforcement Learning28 Feb
  9. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  10. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  11. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  12. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
11 more
  1. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  2. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  5. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  6. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  7. 2020Critic Regularized Regression26 Jun
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 13

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2017Constrained Policy Optimizationcited by 928
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  7. 2021Implicit Behavioral Cloningcited by 442
  8. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  9. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  10. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  11. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learningcited by 41
  12. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
1 more
  1. 2025Diffusion Self-Weighted Guidance for Offline Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning
  2. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
  3. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
  4. 2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation
  5. 2026Robust Regularized Policy Iteration under Transition Uncertainty
  6. 2025From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learning
  7. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning
  8. 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning