Data to 7 Oct 2026

Offline Reinforcement Learning with Fisher Divergence Critic Regularization

arXiv 2103.08050 ↗First posted 14 Mar 2021cs.LGcited by 237

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Many modern approaches to offline Reinforcement Learning (RL) utilize behavior regularization, typically augmenting a model-free actor critic algorithm with a penalty measuring divergence of the policy from the offline data. In this work, we propose an alternative approach to encouraging the learned policy to stay close to the data, namely parameterizing the critic as the log-behavior-policy, which generated the offline data, plus a state-action value offset term, which can be learned using a neural network. Behavior regularization then corresponds to an appropriate regularizer on the offset term. We propose using a gradient penalty regularizer for the offset term and demonstrate its equivalence to Fisher divergence regularization, suggesting connections to the score matching and generative energy-based model literature. We thus term our resulting algorithm Fisher-BRC (Behavior Regularized Critic). On standard offline RL benchmarks, Fisher-BRC achieves both improved performance and faster convergence over existing state-of-the-art methods.

Key facts. Cited by 237 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 34 references that are in the map. First posted on arXiv 14 Mar 2021.

Builds on

24 of 34 references.

Closest ancestors · 5

Same direction, newest first

Other references · 19

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2015Continuous control with deep reinforcement learningcited by 3,279
2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
2020Why Normalizing Flows Fail to Detect Out-of-Distribution Datacited by 220
2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178

7 more

2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
2020Model-Based Offline Planningcited by 126
2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
2018Image-to-image translation for cross-domain disentanglementcited by 61
2020Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 64; 2023: 55; 2024: 56; 2025: 28; 2026: 12.

237 papers.

Sort:

By area:

2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
2021Implicit Behavioral Cloningcited by 442
2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
2022Behavior Transformers: Cloning k modes with one stonecited by 299
2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
2022Online Decision Transformercited by 194
2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
2021Offline RL Without Off-Policy Evaluationcited by 150

38 more

2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
2022Supported Policy Optimization for Offline Reinforcement Learningcited by 73
2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
2023Score Regularized Policy Optimization through Diffusion Behaviorcited by 67
2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 59
2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 59
2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
2022Offline RL Policies Should be Trained to be Adaptivecited by 52
2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
2022Bootstrapped Transformer for Offline Reinforcement Learningcited by 37
2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 34
2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learningcited by 33
2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 31

50 most cited of 237 All 237 (JSON)

Similar, not cited

Close in wording, no citation either way