mlsanityData to 14 Oct 2026

BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning

arXiv 1910.12179 ↗First posted 27 Oct 2019cs.LG cs.AI stat.MLcited by 91

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

There has recently been a surge in research in batch Deep Reinforcement Learning (DRL), which aims for learning a high-performing policy from a given dataset without additional interactions with the environment. We propose a new algorithm, Best-Action Imitation Learning (BAIL), which strives for both simplicity and performance. BAIL learns a V function, uses the V function to select actions it believes to be high-performing, and then uses those actions to train a policy network using imitation learning. For the MuJoCo benchmark, we provide a comprehensive experimental study of BAIL, comparing its performance to four other batch Q-learning and imitation-learning schemes for a large variety of batch datasets. Our experiments show that BAIL's performance is much higher than the other schemes, and is also computationally much faster than the batch Q-learning schemes.

Key facts. Cited by 91 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 52 references that are in the map. First posted on arXiv 27 Oct 2019.

Builds on

26 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning29 May
  2. 2018Fast Policy Learning through Imitation and Reinforcement26 May
  3. 2018Reinforcement Learning from Imperfect Demonstrations14 Feb

Other references · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  12. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
11 more
  1. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  2. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  3. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  4. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  5. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 111
  6. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  7. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  8. 2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
  9. 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
  10. 2018Supervised Policy Update for Deep Reinforcement Learningcited by 11
  11. 2019Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Samplingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 2; 2021: 11; 2022: 19; 2023: 21; 2024: 21; 2025: 11; 2026: 6.

91 papers.

Sort:

By area:

  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  2. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  3. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  4. 2021Offline RL Without Off-Policy Evaluationcited by 151
  5. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  6. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  7. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  8. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  9. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  10. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  11. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 69
  12. 2022A Survey on Model-based Reinforcement Learningcited by 66
38 more
  1. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  2. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  3. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 52
  4. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  5. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 41
  6. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  7. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 32
  8. 2024GeRM: A Generalist Robotic Model with Mixture-of-experts for Quadruped Robotcited by 29
  9. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learningcited by 28
  10. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  11. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learningcited by 26
  12. 2020Semi-supervised reward learning for offline reinforcement learningcited by 23
  13. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  14. 2022Boosting Offline Reinforcement Learning via Data Rebalancingcited by 22
  15. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  16. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  17. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  18. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 21
  19. 2023Supported Trust Region Optimization for Offline Reinforcement Learningcited by 20
  20. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimizationcited by 19
  21. 2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applicationscited by 17
  22. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  23. 2024Efficacy of Language Model Self-Play in Non-Zero-Sum Gamescited by 15
  24. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 15
  25. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidancecited by 13
  26. 2022User-Interactive Offline Reinforcement Learningcited by 11
  27. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 11
  28. 2021Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learningcited by 10
  29. 2022BATS: Best Action Trajectory Stitchingcited by 10
  30. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learningcited by 10
  31. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learningcited by 9
  32. 2022Model-based trajectory stitching for improved behavioural cloning and its applicationscited by 9
  33. 2023Pre-training with Synthetic Data Helps Offline Reinforcement Learningcited by 9
  34. 2023Improving Behavioural Cloning with Positive Unlabeled Learningcited by 8
  35. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  36. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  37. 2024Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defensescited by 7
  38. 2021Quantile Filtered Imitation Learningcited by 6

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Strong Baseline for Batch Imitation Learning
  2. 2025What Matters for Batch Online Reinforcement Learning in Robotics?
  3. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions
  4. 2022Planning for Sample Efficient Imitation Learning
  5. 2021A Pragmatic Look at Deep Imitation Learning
  6. 2019Few-Shot Bayesian Imitation Learning with Logical Program Policies
  7. 2019A Divergence Minimization Perspective on Imitation Learning Methods
  8. 2024VITAL: Interactive Few-Shot Imitation Learning via Visual Human-in-the-Loop Corrections