mlsanityData to 7 Oct 2026

Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data

arXiv 2210.08642 ↗First posted 16 Oct 2022cs.LG cs.AI cs.ROcited by 12

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) can be used to improve future performance by leveraging historical data. There exist many different algorithms for offline RL, and it is well recognized that these algorithms, and their hyperparameter settings, can lead to decision policies with substantially differing performance. This prompts the need for pipelines that allow practitioners to systematically perform algorithm-hyperparameter selection for their setting. Critically, in most real-world settings, this pipeline must only involve the use of historical data. Inspired by statistical model selection methods for supervised learning, we introduce a task- and method-agnostic pipeline for automatically training, comparing, selecting, and deploying the best policy when the provided dataset is limited in size. In particular, our work highlights the importance of performing multiple data splits to produce more reliable algorithm-hyperparameter selection. While this is a common approach in supervised learning, to our knowledge, this has not been discussed in detail in the offline RL setting. We show it can have substantial impacts when the dataset is small. Compared to alternate approaches, our proposed pipeline outputs higher-performing deployed policies from a broad range of offline policy learning algorithms and across various simulation domains in healthcare, education, and robotics. This work contributes toward the development of a general-purpose meta-algorithm for automatic algorithm-hyperparameter selection for offline RL.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 70 references that are in the map. First posted on arXiv 16 Oct 2022.

Builds on

33 of 70 references.

Closest ancestors · 12

Same direction, newest first

  1. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning26 Oct
  2. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  3. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction2 Feb
  4. 2020Batch Value-function Approximation with Only Realizability11 Aug
  5. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  6. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 21

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  3. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  4. 2019Batch Policy Learning under Constraintscited by 242
  5. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  6. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  7. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  8. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  9. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 114
  10. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  11. 2019Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activitycited by 70
  12. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
9 more
  1. 2018Policy Optimization via Importance Samplingcited by 47
  2. 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
  3. 2016Double/Debiased Machine Learning for Treatment and Causal Parameterscited by 41
  4. 2021Heuristic-Guided Reinforcement Learningcited by 41
  5. 2020Adaptive Estimator Selection for Off-Policy Evaluationcited by 38
  6. 2020POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learningcited by 36
  7. 2020Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Makingcited by 30
  8. 2021Model Selection in Batch Policy Optimizationcited by 11
  9. 2021Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 3; 2025: 2; 2026: 1.

12 papers.

  1. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
  2. 2023Reinforcement Learning Tutor Better Supported Lower Performers in a Math Taskcited by 11
  3. 2024Preference Aligned Diffusion Planner for Quadrupedal Locomotion Controlcited by 7
  4. 2023Off-Policy Evaluation for Human Feedbackcited by 6
  5. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocolcited by 6
  6. 2023Revisiting Bellman Errors for Offline Model Selectioncited by 5
  7. 2023HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcarecited by 5
  8. 2023When is Offline Policy Selection Sample Efficient for Reinforcement Learning?cited by 3
  9. 2024Cross-Validated Off-Policy Evaluationcited by 3
  10. 2025Alert: Learning Trigger Functions for Early Classification of Time Series using Deep-RLcited by 1
  11. 2024Off-Policy Selection for Initiating Human-Centric Experimental Designcited by 0
  12. 2026Towards Scalable Multi-Task Reinforcement Learning with Large Decision Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A Minimalist Approach to Offline Reinforcement Learning
  2. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  3. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  4. 2025A Clean Slate for Offline Reinforcement Learning
  5. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  6. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning
  7. 2025Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset
  8. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms