Data to 7 Oct 2026

RORL: Robust Offline Reinforcement Learning via Conservative Smoothing

arXiv 2206.02829 ↗First posted 6 Jun 2022cs.LG cs.AI stat.MLcited by 83

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) provides a promising direction to exploit massive amount of offline data for complex decision-making tasks. Due to the distribution shift issue, current offline RL algorithms are generally designed to be conservative in value estimation and action selection. However, such conservatism can impair the robustness of learned policies when encountering observation deviation under realistic conditions, such as sensor errors and adversarial attacks. To trade off robustness and conservatism, we propose Robust Offline Reinforcement Learning (RORL) with a novel conservative smoothing technique. In RORL, we explicitly introduce regularization on the policy and the value function for states near the dataset, as well as additional conservative value estimation on these states. Theoretically, we show RORL enjoys a tighter suboptimality bound than recent theoretical results in linear MDPs. We demonstrate that RORL can achieve state-of-the-art performance on the general offline RL benchmark and is considerably robust to adversarial observation perturbations.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 58 of its 82 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

58 of 82 references.

Closest ancestors · 24

Same direction, newest first

12 more

2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
2020Is Pessimism Provably Efficient for Offline RL?30 Dec
2020Critic Regularized Regression26 Jun
2020On Reward-Free Reinforcement Learning with Linear Function Approximation19 Jun
2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
2020MOPO: Model-based Offline Policy Optimization27 May
2020MOReL : Model-Based Offline Reinforcement Learning12 May
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
2019Behavior Regularized Offline Reinforcement Learning26 Nov
2019Policy Poisoning in Batch Reinforcement Learning and Control13 Oct

Other references · 34

2014Explaining and Harnessing Adversarial Examplescited by 7,287
2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
2016Deep Exploration via Bootstrapped DQNcited by 518
2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
2017Minimax Regret Bounds for Reinforcement Learningcited by 500
2017Robust Adversarial Reinforcement Learningcited by 439
2017Adversarial Attacks on Neural Network Policiescited by 314

22 more

2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
2020First return, then explorecited by 256
2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 209
2019Active Domain Randomizationcited by 161
2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 152
2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
2017Vulnerability of Deep Reinforcement Learning to Policy Induction Attackscited by 91
2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
2020Robust Reinforcement Learning using Adversarial Populationscited by 64
2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
2017Reinforcement Learning under Model Mismatchcited by 52
2020Robust Reinforcement Learning via Adversarial training with Langevin Dynamicscited by 45
2021Robust Policy Gradient against Strong Data Corruptioncited by 30
2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
2021Dynamic Bottleneck for Robust Self-Supervised Explorationcited by 21
2013Scaling Up Robust MDPs by Reinforcement Learningcited by 16
2021Safe Exploration by Solving Early Terminated MDPcited by 13
2020FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularizationcited by 6
2022Policy Learning for Robust Markov Decision Process with a Mismatched Generative Modelcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 19; 2024: 28; 2025: 22; 2026: 8.

83 papers.

Sort:

By area:

2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
2023Anti-Exploration by Random Network Distillationcited by 49
2023Behavior Proximal Policy Optimizationcited by 43
2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learningcited by 33
2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
2022First-order Policy Optimization for Robust Markov Decision Processcited by 31
2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 29

38 more

2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
2025RobustVLA: On Robustness of Vision-Language-Action Model against Multi-Modal Perturbationscited by 23
2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 21
2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
2023Survival Instinct in Offline Reinforcement Learningcited by 20
2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppressioncited by 17
2024Refining Minimax Regret for Unsupervised Environment Designcited by 16
2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operatorscited by 14
2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learningcited by 13
2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learningcited by 12
2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operatorcited by 9
2024Exclusively Penalized Q-learning for Offline Reinforcement Learningcited by 8
2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
2022Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shapingcited by 6
2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replaycited by 6
2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learningcited by 6
2025Habitizing Diffusion Planning for Efficient and Effective Decision Makingcited by 6
2026Manifold-Constrained Energy-Based Transition Models for Offline Reinforcement Learningcited by 6
2022What is Flagged in Uncertainty Quantification? Latent Density Models for Uncertainty Categorizationcited by 5
2024Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defensescited by 5
2024Offline Reinforcement Learning from Datasets with Structured Non-Stationaritycited by 5
2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learningcited by 4
2024Corruption-Robust Offline Two-Player Zero-Sum Markov Gamescited by 4
2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?cited by 4
2024Mitigating Adversarial Perturbations for Deep Reinforcement Learning via Vector Quantizationcited by 4
2024Distilling LLMs' Decomposition Abilities into Compact Language Modelscited by 3
2024Robust RL with LLM-Driven Data Synthesis and Policy Adaptation for Autonomous Drivingcited by 3
2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spacescited by 3

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way