Data to 7 Oct 2026

A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems

arXiv 2203.01387 ↗First posted 2 Mar 2022cs.LG cs.AI stat.MLcited by 228

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

With the widespread adoption of deep learning, reinforcement learning (RL) has experienced a dramatic increase in popularity, scaling to previously intractable problems, such as playing complex games from pixel observations, sustaining conversations with humans, and controlling robotic agents. However, there is still a wide range of domains inaccessible to RL due to the high cost and danger of interacting with the environment. Offline RL is a paradigm that learns exclusively from static datasets of previously collected interactions, making it feasible to extract policies from large and diverse training datasets. Effective offline RL algorithms have a much wider range of applications than online RL, being particularly appealing for real-world applications, such as education, healthcare, and robotics. In this work, we contribute with a unifying taxonomy to classify offline RL methods. Furthermore, we provide a comprehensive review of the latest algorithmic breakthroughs in the field using a unified notation as well as a review of existing benchmarks' properties and shortcomings. Additionally, we provide a figure that summarizes the performance of each method and class of methods on different dataset properties, equipping researchers with the tools to decide which type of algorithm is best suited for the problem at hand and identify which classes of algorithms look the most promising. Finally, we provide our perspective on open problems and propose future research directions for this rapidly growing field.

Key facts. Cited by 228 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 62 of its 100 references that are in the map. First posted on arXiv 2 Mar 2022.

Builds on

62 of 100 references.

Closest ancestors · 21

Same direction, newest first

9 more

2020Critic Regularized Regression26 Jun
2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
2020MOPO: Model-based Offline Policy Optimization27 May
2020MOReL : Model-Based Offline Reinforcement Learning12 May
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
2019Behavior Regularized Offline Reinforcement Learning26 Nov
2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 41

2017Attention Is All You Needcited by 59,932
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2015Trust Region Policy Optimizationcited by 2,503
2017CARLA: An Open Urban Driving Simulatorcited by 2,431
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
2018DeepMind Control Suitecited by 935
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753

29 more

2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
2017A Brief Survey of Deep Reinforcement Learningcited by 466
2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
2020dm_control: Software and Tasks for Continuous Controlcited by 368
2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
2018Maximum a Posteriori Policy Optimisationcited by 318
2019Challenges of Real-World Reinforcement Learningcited by 282
2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
2014Algorithms for CVaR Optimization in MDPscited by 98
2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
2021Benchmarks for Deep Off-Policy Evaluationcited by 89
2021Conservative Objective Models for Effective Offline Model-Based Optimizationcited by 87
2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
2020An empirical investigation of the challenges of real-world reinforcement learningcited by 75
2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
2017Consistent On-Line Off-Policy Evaluationcited by 51
2019AlphaStar: An Evolutionary Computation Perspectivecited by 50
2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 47
2021Reinforcement Learning for Education: Opportunities and Challengescited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 47; 2024: 58; 2025: 65; 2026: 45.

228 papers.

Sort:

By area:

2022Active Example Selection for In-Context Learningcited by 195
2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
2023OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensemblingcited by 71
2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
2024Cost-Effective Online Multi-LLM Selection with Versatile Reward Modelscited by 36
2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35

38 more

2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitchingcited by 35
2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learningcited by 33
2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
2023The Generalization Gap in Offline Reinforcement Learningcited by 25
2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19
2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
2022Dungeons and Data: A Large-Scale NetHack Datasetcited by 16
2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Modelscited by 15
2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robotscited by 15
2023IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Controlcited by 14
2024CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planningcited by 14
2025Offline Model-Based Optimization: Comprehensive Reviewcited by 14
2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidancecited by 13
2024Learning from Imperfect Demonstrations with Self-Supervision for Robotic Manipulationcited by 13
2022Federated Learning and Meta Learning: Approaches, Applications, and Directionscited by 12
2023A Survey of Demonstration Learningcited by 12
2023Boosting Offline Reinforcement Learning with Action Preference Querycited by 12
2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
2023ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learningcited by 11
2023H-GAP: Humanoid Control with a Generalist Plannercited by 11
2023Learning to View: Decision Transformers for Active Object Detectioncited by 10
2023Learning to Reach Goals via Diffusioncited by 10
2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 10
2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RLcited by 10
2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
2025A Clean Slate for Offline Reinforcement Learningcited by 10
2024Learning from Sparse Offline Datasets via Conservative Density Estimationcited by 9
2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluationcited by 9
2025A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Controlcited by 9
2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
2023Minimax Optimal Q Learning with Nearest Neighborscited by 8
2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Modelscited by 8
2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8

50 most cited of 228 All 228 (JSON)

Similar, not cited

Close in wording, no citation either way