mlsanityData to 14 Oct 2026

Policy Regularization with Dataset Constraint for Offline Reinforcement Learning

arXiv 2306.06569 ↗First posted 11 Jun 2023cs.LG cs.AIcited by 34

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We consider the problem of learning the best possible policy from a fixed dataset, known as offline Reinforcement Learning (RL). A common taxonomy of existing offline RL works is policy regularization, which typically constrains the learned policy by distribution or support of the behavior policy. However, distribution and support constraints are overly conservative since they both force the policy to choose similar actions as the behavior policy when considering particular states. It will limit the learned policy's performance, especially when the behavior policy is sub-optimal. In this paper, we find that regularizing the policy towards the nearest state-action pair can be more effective and thus propose Policy Regularization with Dataset Constraint (PRDC). When updating the policy in a given state, PRDC searches the entire dataset for the nearest state-action sample and then restricts the policy with the action of this sample. Unlike previous works, PRDC can guide the policy with proper behaviors from the dataset, allowing it to choose actions that do not appear in the dataset along with the given state. It is a softer constraint but still keeps enough conservatism from out-of-distribution actions. Empirical evidence and theoretical analysis show that PRDC can alleviate offline RL's fundamentally challenging value overestimation issue with a bounded performance gap. Moreover, on a set of locomotion and navigation tasks, PRDC achieves state-of-the-art performance compared with existing methods. Code is available at https://github.com/LAMDA-RL/PRDC

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 46 references that are in the map. First posted on arXiv 11 Jun 2023.

Builds on

27 of 46 references.

Closest ancestors · 16

Same direction, newest first

  1. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  2. 2022Large-Scale Retrieval for Reinforcement Learning10 Jun
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  4. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  6. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  11. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  12. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
4 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 11

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2016OpenAI Gymcited by 1,736
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2016An overview of gradient descent optimization algorithmscited by 1,033
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  9. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  10. 2021Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robotscited by 116
  11. 2022Retrieval-Augmented Reinforcement Learningcited by 53

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 12; 2026: 11.

34 papers.

Sort:

By area:

  1. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  2. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learningcited by 20
  3. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 17
  4. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  5. 2025Device-Cloud Collaborative LLM Inference with Multi-Modal, Multi-Task, Multi-Turn Conversationscited by 14
  6. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learningcited by 13
  7. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 11
  8. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  9. 2025Bootstrap Off-policy with World Modelcited by 7
  10. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replaycited by 6
  11. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 6
  12. 2023HiBid: A Cross-Channel Constrained Bidding System with Budget Allocation by Hierarchical Offline Deep Reinforcement Learningcited by 4
22 more
  1. 2026Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Actioncited by 4
  2. 2025Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhoodcited by 3
  3. 2024Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamicscited by 2
  4. 2024Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learningcited by 2
  5. 2025ImagineBench: Evaluating Reinforcement Learning with Large Language Model Rolloutscited by 2
  6. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learningcited by 2
  7. 2025Guardian-regularized Safe Offline Reinforcement Learning for Smart Weaning of Mechanical Circulatory Devicescited by 2
  8. 2024SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasetscited by 1
  9. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learningcited by 1
  10. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RLcited by 1
  11. 2026VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agentscited by 1
  12. 2026Beyond Penalization: Diffusion-based Out-of-Distribution Detection and Selective Regularization in Offline Reinforcement Learningcited by 1
  13. 2024Hypercube Policy Regularization Framework for Offline Reinforcement Learningcited by 0
  14. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learningcited by 0
  15. 2026CDRL: A Reinforcement Learning Framework Inspired by Cerebellar Circuits and Dendritic Computational Strategiescited by 0
  16. 2026Fog of Love: Engineering Virtuous Agent Behavior with Affinity-based Reinforcement Learning in a Game Environmentcited by 0
  17. 2026Expert Behavior Prior Reinforcement Learningcited by 0
  18. 2026Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learningcited by 0
  19. 2026DiDrive: A Risk-Aware Hierarchical Diffusion Framework for Safe Offline Reinforcement Learning in Autonomous Drivingcited by 0
  20. 2026Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transportcited by 0
  21. 2026Improving Online Reinforcement Learning via Bidirectional Behavior Prior Distillationcited by 0
  22. 2026From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  2. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  3. 2021Behavior Constraining in Weight Space for Offline Reinforcement Learning
  4. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  5. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learning
  6. 2020Critic Regularized Regression
  7. 2026Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
  8. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL