mlsanityData to 14 Oct 2026

Offline Reinforcement Learning with Adaptive Behavior Regularization

arXiv 2211.08251 ↗First posted 15 Nov 2022cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) defines a sample-efficient learning paradigm, where a policy is learned from static and previously collected datasets without additional interaction with the environment. The major obstacle to offline RL is the estimation error arising from evaluating the value of out-of-distribution actions. To tackle this problem, most existing offline RL methods attempt to acquire a policy both ``close" to the behaviors contained in the dataset and sufficiently improved over them, which requires a trade-off between two possibly conflicting targets. In this paper, we propose a novel approach, which we refer to as adaptive behavior regularization (ABR), to balance this critical trade-off. By simply utilizing a sample-based regularization, ABR enables the policy to adaptively adjust its optimization objective between cloning and improving over the policy used to generate the dataset. In the evaluation on D4RL datasets, a widely adopted benchmark for offline reinforcement learning, ABR can achieve improved or competitive performance compared to existing state-of-the-art algorithms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 34 references that are in the map. First posted on arXiv 15 Nov 2022.

Builds on

27 of 34 references.

Closest ancestors · 11

Same direction, newest first

  1. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  6. 2020Critic Regularized Regression26 Jun
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 16

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  9. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  12. 2016Deep Exploration via Bootstrapped DQNcited by 527
4 more
  1. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  2. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  3. 2017Advances in Variational Inferencecited by 236
  4. 2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  2. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
  3. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
  4. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  5. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  6. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  7. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  8. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning