mlsanityData to 14 Oct 2026

TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasets

arXiv 2212.02125 ↗First posted 5 Dec 2022stat.ML cs.AI cs.LGcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We consider an offline reinforcement learning (RL) setting where the agent need to learn from a dataset collected by rolling out multiple behavior policies. There are two challenges for this setting: 1) The optimal trade-off between optimizing the RL signal and the behavior cloning (BC) signal changes on different states due to the variation of the action coverage induced by different behavior policies. Previous methods fail to handle this by only controlling the global trade-off. 2) For a given state, the action distribution generated by different behavior policies may have multiple modes. The BC regularizers in many previous methods are mean-seeking, resulting in policies that select out-of-distribution (OOD) actions in the middle of the modes. In this paper, we address both challenges by using adaptively weighted reverse Kullback-Leibler (KL) divergence as the BC regularizer based on the TD3 algorithm. Our method not only trades off the RL and BC signals with per-state weights (i.e., strong BC regularization on the states with narrow action coverage, and vice versa) but also avoids selecting OOD actions thanks to the mode-seeking property of reverse KL. Empirically, our algorithm can outperform existing offline RL algorithms in the MuJoCo locomotion tasks with the standard D4RL datasets as well as the mixed datasets that combine the standard datasets.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 44 references that are in the map. First posted on arXiv 5 Dec 2022.

Builds on

33 of 44 references.

Closest ancestors · 14

Same direction, newest first

  1. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  2. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  4. 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
  5. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  6. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  7. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
2 more
  1. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  2. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 19

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  3. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  8. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  9. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  10. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 115
  11. 2019Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustnesscited by 104
  12. 2020BADGR: An Autonomous Self-Supervised Learning-Based Navigation Systemcited by 102
7 more
  1. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  2. 2021Adversarially Guided Actor-Criticcited by 47
  3. 2020Inverse Constrained Reinforcement Learningcited by 46
  4. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 41
  5. 2021Cross DQN: Cross Deep Q Network for Ads Allocation in Feedcited by 19
  6. 2019Safe Policy Improvement with an Estimated Baseline Policycited by 17
  7. 2019Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environmentscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 1.

4 papers.

  1. 2024Learning on One Mode: Addressing Multi-modality in Offline Reinforcement Learningcited by 3
  2. 2023Mildly Constrained Evaluation Policy for Offline Reinforcement Learningcited by 1
  3. 2025Reward-Driven Interaction: Enhancing Proactive Dialogue Agents through User Satisfaction Predictioncited by 0
  4. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasetscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuning
  2. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  3. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  4. 2021BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning
  5. 2025B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learning
  6. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
  7. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning
  8. 2024Policy-regularized Offline Multi-objective Reinforcement Learning