mlsanityData to 14 Oct 2026

Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning

arXiv 2508.19900 ↗First posted 27 Aug 2025cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) enables learning effective policies from fixed datasets without any environment interaction. Existing methods typically employ policy constraints to mitigate the distribution shift encountered during offline RL training. However, because the scale of the constraints varies across tasks and datasets of differing quality, existing methods must meticulously tune hyperparameters to match each dataset, which is time-consuming and often impractical. We propose Adaptive Scaling of Policy Constraints (ASPC), a second-order differentiable framework that dynamically balances RL and behavior cloning (BC) during training. We theoretically analyze its performance improvement guarantee. In experiments on 39 datasets across four D4RL domains, ASPC using a single hyperparameter configuration outperforms other adaptive constraint methods and state-of-the-art offline RL algorithms that require per-dataset tuning while incurring only minimal computational overhead. The code will be released at https://github.com/Colin-Jing/ASPC.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 44 references that are in the map. First posted on arXiv 27 Aug 2025.

Builds on

36 of 44 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning30 May
  2. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance4 Sep
  3. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting22 Jun
  4. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning11 Jun
  5. 2023Boosting Offline Reinforcement Learning with Action Preference Query6 Jun
  6. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  7. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learning10 Apr
  8. 2023Anti-Exploration by Random Network Distillation31 Jan
  9. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes28 Nov
  10. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  11. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  12. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
8 more
  1. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  7. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  8. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  9. 2018Learning to Drive in a Daycited by 273
  10. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  11. 2022Multi-Game Decision Transformerscited by 214
  12. 2025Flow Q-Learningcited by 184
4 more
  1. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  2. 2017A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Unitscited by 50
  3. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 48
  4. 2022TorchOpt: An Efficient Library for Differentiable Optimizationcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Role-Adaptive Policy Optimization for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Clean Slate for Offline Reinforcement Learning
  2. 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning
  3. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
  4. 2026Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
  5. 2025From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learning
  6. 2020Hyperparameter Selection for Offline Reinforcement Learning
  7. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  8. 2025Sample-Efficient Policy Constraint Offline Deep Reinforcement Learning based on Sample Filtering