mlsanityData to 14 Oct 2026

Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters

arXiv 2110.04156 ↗First posted 8 Oct 2021cs.LG cs.AIcited by 24

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In this work, we argue for the importance of an online evaluation budget for a reliable comparison of deep offline RL algorithms. First, we delineate that the online evaluation budget is problem-dependent, where some problems allow for less but others for more. And second, we demonstrate that the preference between algorithms is budget-dependent across a diverse range of decision-making domains such as Robotics, Finance, and Energy Management. Following the points above, we suggest reporting the performance of deep offline RL algorithms under varying online evaluation budgets. To facilitate this, we propose to use a reporting tool from the NLP field, Expected Validation Performance. This technique makes it possible to reliably estimate expected maximum performance under different budgets while not requiring any additional computation beyond hyperparameter search. By employing this tool, we also show that Behavioral Cloning is often more favorable to offline RL algorithms when working within a limited budget.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 53 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

40 of 53 references.

Closest ancestors · 14

Same direction, newest first

  1. 2021Offline RL Without Off-Policy Evaluation16 Jun
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  4. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  5. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  6. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  7. 2020Critic Regularized Regression26 Jun
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
2 more
  1. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  2. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 26

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2018Model Cards for Model Reportingcited by 1,080
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  7. 2017Deep Reinforcement Learning that Matterscited by 754
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  9. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  10. 2019Batch Policy Learning under Constraintscited by 244
  11. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  12. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 169
14 more
  1. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  2. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  3. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  4. 2020FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Financecited by 87
  5. 2020Self-Supervised Reinforcement Learning for Recommender Systemscited by 49
  6. 2018Horizon: Facebook's Open Source Applied Reinforcement Learning Platformcited by 48
  7. 2021A Deep Value-network Based Approach for Multi-Driver Order Dispatchingcited by 45
  8. 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
  9. 2017A Benchmark Environment Motivated by Industrial Control Problemscited by 31
  10. 2020Offline Policy Selection under Uncertaintycited by 31
  11. 2021Hyperparameter Selection for Imitation Learningcited by 17
  12. 2021Expected Validation Performance and Estimation of a Random Variable's Maximumcited by 6
  13. 2020Showing Your Work Doesn't Always Workcited by 5
  14. 2021MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 7; 2024: 5; 2025: 1; 2026: 5.

24 papers.

Sort:

By area:

  1. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  2. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
  3. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 89
  4. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 74
  5. 2023Anti-Exploration by Random Network Distillationcited by 50
  6. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 33
  7. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  8. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 21
  9. 2023Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluationcited by 13
  10. 2023A Survey of Demonstration Learningcited by 12
  11. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learningcited by 12
  12. 2022User-Interactive Offline Reinforcement Learningcited by 11
12 more
  1. 2025A Clean Slate for Offline Reinforcement Learningcited by 11
  2. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluationcited by 9
  3. 2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learningcited by 8
  4. 2024N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needscited by 7
  5. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6
  6. 2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?cited by 4
  7. 2022Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learningcited by 3
  8. 2026Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learningcited by 1
  9. 2026Neuro-Inspired Inverse Learning for Planning and Controlcited by 0
  10. 2026Generative OOD-regularized Model-based Policy Optimizationcited by 0
  11. 2026Active Offline-to-Online Reinforcement Learningcited by 0
  12. 2026CoDrift: Compositional Drifting for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  2. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  3. 2020Offline Reinforcement Learning Hands-On
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  5. 2024Is Value Learning Really the Main Bottleneck in Offline RL?
  6. 2021Instabilities of Offline RL with Pre-Trained Neural Representation
  7. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  8. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning