mlsanityData to 14 Oct 2026

JumpStart Your Policy Learning with Lessons from 160,000 Training Runs

arXiv 2609.13730 ↗First posted 12 Sep 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Reliable progress in offline policy learning depends on careful reporting, well-tuned baselines, and evaluation across diverse conditions. Prior work has shown that results can be sensitive to reporting choices, hyperparameter tuning, and dataset properties, but these sources of variability have not been systematically investigated together at the scale needed to understand how they shape conclusions. To address this gap, we present a large-scale empirical study of offline reinforcement and imitation learning, training over 160,000 policies across 114 datasets. At this scale, no algorithm dominates: aggregate performance among the strongest methods is often close, but the leaders differ substantially across environments. We find that proper hyperparameter tuning frequently reshuffles perceived algorithm rankings and that benchmark composition can produce conflicting conclusions. We also study hyperparameter sensitivity and transfer across environments, identifying a simple strategy for deriving strong default configurations. We use our findings to develop a dataset-conditioned recommender that provides task-specific algorithm recommendations for practitioners. Finally, we release JumpStart: a resource suite containing every trained policy, per-model scores and hyperparameters, strong baselines across all environments, training and evaluation code, and an extensible website for retrieving, analyzing, and contributing results. Together, these resources aim to make offline policy-learning research more reliable and enable future work beyond the scope of this study.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 45 references that are in the map. First posted on arXiv 12 Sep 2026.

Builds on

35 of 45 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning15 Aug
  2. 2024Simple Ingredients for Offline Reinforcement Learning19 Mar
  3. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms1 Jun
  4. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  5. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  6. 2021A Dataset Perspective on Offline Reinforcement Learning8 Nov
  7. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  4. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  6. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  8. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  12. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
11 more
  1. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  2. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  3. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  4. 2019Model-Based Reinforcement Learning for Ataricited by 502
  5. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  6. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  7. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  8. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
  9. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  10. 2021What Matters for Adversarial Imitation Learning?cited by 70
  11. 2023When should we prefer Decision Transformers for Offline Reinforcement Learning?cited by 20

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Offline Behavioral Data Selection
  2. 2026Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
  3. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  4. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning
  5. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  6. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data
  7. 2025A Clean Slate for Offline Reinforcement Learning
  8. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL