mlsanityData to 14 Oct 2026

Offline Reinforcement Learning with Soft Behavior Regularization

arXiv 2110.07395 ↗First posted 14 Oct 2021cs.LGcited by 28

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Most prior approaches to offline reinforcement learning (RL) utilize textitbehavior regularization, typically augmenting existing off-policy actor critic algorithms with a penalty measuring divergence between the policy and the offline data. However, these approaches lack guaranteed performance improvement over the behavior policy. In this work, we start from the performance difference between the learned policy and the behavior policy, we derive a new policy learning objective that can be used in the offline setting, which corresponds to the advantage function value of the behavior policy, multiplying by a state-marginal density ratio. We propose a practical way to compute the density ratio and demonstrate its equivalence to a state-dependent behavior regularization. Unlike state-independent regularization used in prior approaches, this textitsoft regularization allows more freedom of policy deviation at high confidence states, leading to better performance and stability. We thus term our resulting algorithm Soft Behavior-regularized Actor Critic (SBAC). Our experimental results show that SBAC matches or outperforms the state-of-the-art on a set of continuous control locomotion and manipulation tasks.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 57 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

44 of 57 references.

Closest ancestors · 14

Same direction, newest first

  1. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  2. 2021Offline RL Without Off-Policy Evaluation16 Jun
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  5. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  6. 2021Offline Reinforcement Learning with Pseudometric Learning2 Mar
  7. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  8. 2020Critic Regularized Regression26 Jun
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
2 more
  1. 2020BRPO: Batch Residual Policy Optimization8 Feb
  2. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2017Constrained Policy Optimizationcited by 983
  9. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  12. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
18 more
  1. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  2. 2016Kernel Mean Embedding of Distributions: A Review and Beyondcited by 364
  3. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  4. 2019Batch Policy Learning under Constraintscited by 244
  5. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  6. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  7. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  8. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  9. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  10. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  11. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  12. 2019Truly Proximal Policy Optimizationcited by 69
  13. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  14. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 48
  15. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  16. 2020Constrained Markov Decision Processes via Backward Value Functionscited by 33
  17. 2020Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learningcited by 16
  18. 2018Reinforcement Learning based Recommender System using Biclustering Techniquecited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 13; 2024: 6; 2025: 1; 2026: 1.

28 papers.

Sort:

By area:

  1. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  2. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 74
  3. 2023Score Regularized Policy Optimization through Diffusion Behaviorcited by 71
  4. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 60
  5. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  6. 2023Behavior Proximal Policy Optimizationcited by 43
  7. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 43
  8. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  9. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  10. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 29
  11. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  12. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
16 more
  1. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  2. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  3. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  4. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Controlcited by 12
  5. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learningcited by 10
  6. 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulationcited by 9
  7. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  8. 2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6
  9. 2023Environment Transformer and Policy Optimization for Model-Based Offline Reinforcement Learningcited by 4
  10. 2023Offline Diversity Maximization Under Imitation Constraintscited by 4
  11. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  12. 2023A Fully Data-Driven Approach for Realistic Traffic Signal Control Using Offline Reinforcement Learningcited by 1
  13. 2026Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learningcited by 1
  14. 2022State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learningcited by 0
  15. 2024A Model-Based Approach for Improving Reinforcement Learning Efficiency Leveraging Expert Observationscited by 0
  16. 2024UDQL: Bridging The Gap between MSE Loss and The Optimal Value Function in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning
  2. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  3. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
  4. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
  5. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  6. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learning
  7. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  8. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning