mlsanityData to 7 Oct 2026

Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning

arXiv 2201.05433 ↗First posted 14 Jan 2022cs.LGcited by 9

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) Algorithms are often designed with environments such as MuJoCo in mind, in which the planning horizon is extremely long and no noise exists. We compare model-free, model-based, as well as hybrid offline RL approaches on various industrial benchmark (IB) datasets to test the algorithms in settings closer to real world problems, including complex noise and partially observable states. We find that on the IB, hybrid approaches face severe difficulties and that simpler algorithms, such as rollout based algorithms or model-free algorithms with simpler regularizers perform best on the datasets.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 48 references that are in the map. First posted on arXiv 14 Jan 2022.

Builds on

34 of 48 references.

Closest ancestors · 16

Same direction, newest first

  1. 2021Behavior Constraining in Weight Space for Offline Reinforcement Learning12 Jul
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  4. 2021Risk-Averse Offline Reinforcement Learning10 Feb
  5. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
  6. 2020DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs18 Oct
  7. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  8. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  9. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  10. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  11. 2020Critic Regularized Regression26 Jun
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
4 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 18

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  7. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  8. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  9. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  10. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  11. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  12. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
6 more
  1. 2017Interpretable Policies for Reinforcement Learning by Genetic Programmingcited by 47
  2. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  3. 2017A Benchmark Environment Motivated by Industrial Control Problemscited by 31
  4. 2021Active Offline Policy Selectioncited by 26
  5. 2020Reducing Sampling Error in Batch Temporal Difference Learningcited by 11
  6. 2021Curriculum Offline Imitation Learningcited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 3; 2024: 4.

9 papers.

  1. 2024SymbolicAI: A framework for logic-based approaches combining generative models and solverscited by 15
  2. 2022A Search-Based Testing Approach for Deep Reinforcement Learning Agentscited by 12
  3. 2022User-Interactive Offline Reinforcement Learningcited by 11
  4. 2023Improving Offline RL by Blending Heuristicscited by 10
  5. 2023SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agentscited by 9
  6. 2024Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?cited by 7
  7. 2024Model-based Offline Quantum Reinforcement Learningcited by 4
  8. 2023Learning Control Policies for Variable Objectives from Offline Datacited by 3
  9. 2024Why long model-based rollouts are no reason for bad Q-value estimatescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey on Offline Model-Based Reinforcement Learning
  2. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  3. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  4. 2025A Clean Slate for Offline Reinforcement Learning
  5. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model
  6. 2024Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulators
  7. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  8. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory