mlsanityData to 7 Oct 2026

Hypercube Policy Regularization Framework for Offline Reinforcement Learning

arXiv 2411.04534 ↗First posted 7 Nov 2024cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning has received extensive attention from scholars because it avoids the interaction between the agent and the environment by learning a policy through a static dataset. However, general reinforcement learning methods cannot get satisfactory results in offline reinforcement learning due to the out-of-distribution state actions that the dataset cannot cover during training. To solve this problem, the policy regularization method that tries to directly clone policies used in static datasets has received numerous studies due to its simplicity and effectiveness. However, policy constraint methods make the agent choose the corresponding actions in the static dataset. This type of constraint is usually over-conservative, which results in suboptimal policies, especially in low-quality static datasets. In this paper, a hypercube policy regularization framework is proposed, this method alleviates the constraints of policy constraint methods by allowing the agent to explore the actions corresponding to similar states in the static dataset, which increases the effectiveness of algorithms in low-quality datasets. It was also theoretically demonstrated that the hypercube policy regularization framework can effectively improve the performance of original algorithms. In addition, the hypercube policy regularization framework is combined with TD3-BC and Diffusion-QL for experiments on D4RL datasets which are called TD3-BC-C and Diffusion-QL-C. The experimental results of the score demonstrate that TD3-BC-C and Diffusion-QL-C perform better than state-of-the-art algorithms like IQL, CQL, TD3-BC and Diffusion-QL in most D4RL environments in approximate time.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 32 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

23 of 32 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023Model-based Offline Reinforcement Learning with Count-based Conservatism21 Jul
  2. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning11 Jun
  3. 2023Improving Offline RL by Blending Heuristics1 Jun
  4. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  9. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
1 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 10

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  4. 2018Behavioral Cloning from Observationcited by 482
  5. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  6. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  7. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  8. 2022GoalsEye: Learning High Speed Precision Table Tennis on a Physical Robotcited by 8
  9. 2024SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learningcited by 8
  10. 2024Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policiescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  2. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  3. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
  4. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  5. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learning
  6. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  7. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
  8. 2022Supported Policy Optimization for Offline Reinforcement Learning