mlsanityData to 7 Oct 2026

Offline Reinforcement Learning with On-Policy Q-Function Regularization

arXiv 2307.13824 ↗First posted 25 Jul 2023cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The core challenge of offline reinforcement learning (RL) is dealing with the (potentially catastrophic) extrapolation error induced by the distribution shift between the history dataset and the desired policy. A large portion of prior work tackles this challenge by implicitly/explicitly regularizing the learning policy towards the behavior policy, which is hard to estimate reliably in practice. In this work, we propose to regularize towards the Q-function of the behavior policy instead of the behavior policy itself, under the premise that the Q-function can be estimated more reliably and easily by a SARSA-style estimate and handles the extrapolation error more straightforwardly. We propose two algorithms taking advantage of the estimated Q-function through regularizations, and demonstrate they exhibit strong performance on the D4RL benchmarks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 38 references that are in the map. First posted on arXiv 25 Jul 2023.

Builds on

29 of 38 references.

Closest ancestors · 17

Same direction, newest first

  1. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  2. 2022Behavior Estimation from Multi-Source Data for Offline Reinforcement Learning29 Nov
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Offline RL Without Off-Policy Evaluation16 Jun
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  8. 2021Regularized Behavior Value Estimation17 Mar
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  10. 2021Offline Reinforcement Learning with Pseudometric Learning2 Mar
  11. 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
5 more
  1. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  2. 2020Critic Regularized Regression26 Jun
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 12

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  10. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  11. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  12. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  2. 2024ROER: Regularized Optimal Experience Replaycited by 3
  3. 2024Enhancing Reinforcement Learning Through Guided Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  2. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  4. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  5. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
  6. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
  7. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  8. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning