Data to 7 Oct 2026

Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning

arXiv 2105.08140 ↗First posted 17 May 2021cs.LGcited by 125

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. However, existing Q-learning and actor-critic based off-policy RL algorithms fail when bootstrapping from out-of-distribution (OOD) actions or states. We hypothesize that a key missing ingredient from the existing methods is a proper treatment of uncertainty in the offline setting. We propose Uncertainty Weighted Actor-Critic (UWAC), an algorithm that detects OOD state-action pairs and down-weights their contribution in the training objectives accordingly. Implementation-wise, we adopt a practical and effective dropout-based uncertainty estimation method that introduces very little overhead over existing RL algorithms. Empirically, we observe that UWAC substantially improves model stability during training. In addition, UWAC out-performs existing offline RL methods on a variety of competitive tasks, and achieves significant performance gains over the state-of-the-art baseline on datasets with sparse demonstrations collected from human experts.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 50 references that are in the map. First posted on arXiv 17 May 2021.

Builds on

36 of 50 references.

Closest ancestors · 5

Same direction, newest first

2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
2020MOPO: Model-based Offline Policy Optimization27 May
2020MOReL : Model-Based Offline Reinforcement Learning12 May
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 31

2017Proximal Policy Optimization Algorithmscited by 14,441
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
2013Playing Atari with Deep Reinforcement Learningcited by 3,211
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685

19 more

2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
2016Deep Exploration via Bootstrapped DQNcited by 518
2015Variational Dropout and the Local Reparameterization Trickcited by 489
2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
2017Noisy Networks for Explorationcited by 262
2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
2019Practical Deep Learning with Bayesian Principlescited by 158
2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 68
2017BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systemscited by 30
2018Variational Bayesian dropout: pitfalls and fixescited by 22
2018Randomized Value Functions via Multiplicative Normalizing Flowscited by 15
2020Reinforcement Learning with Uncertainty Estimation for Tactical Decision-Making in Intersectionscited by 15
2017Variational Deep Q Networkcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 28; 2023: 29; 2024: 29; 2025: 13; 2026: 13.

125 papers.

Sort:

By area:

2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 59
2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51

38 more

2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
2022A Review of Uncertainty for Deep Reinforcement Learningcited by 36
2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 31
2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
2022Sample Efficient Deep Reinforcement Learning via Uncertainty Estimationcited by 26
2022Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systemscited by 22
2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
2022Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemblecited by 18
2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applicationscited by 17
2022On the Role of Discount Factor in Offline Reinforcement Learningcited by 16
2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learningcited by 16
2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learningcited by 14
2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
2023Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skillscited by 13
2022Offline Reinforcement Learning with Differentiable Function Approximation is Provably Efficientcited by 12
2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learningcited by 11
2025Uncertainty Quantification for Machine Learning in Healthcare: A Surveycited by 11
2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learningcited by 10
2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarkscited by 10
2021False Correlation Reduction for Offline Reinforcement Learningcited by 9
2023Conservative State Value Estimation for Offline Reinforcement Learningcited by 9
2026TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapsecited by 9
2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environmentscited by 8
2024Exclusively Penalized Q-learning for Offline Reinforcement Learningcited by 8
2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
2022Epistemic Monte Carlo Tree Searchcited by 7

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way