mlsanityData to 7 Oct 2026

Batch Policy Learning under Constraints

arXiv 1903.08738 ↗First posted 20 Mar 2019cs.LG cs.AI math.OCcited by 242

Not on the map. Why?

Cited by 100 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

When learning policies for real-world domains, two important questions arise: (i) how to efficiently use pre-collected off-policy, non-optimal behavior data; and (ii) how to mediate among different competing objectives and constraints. We thus study the problem of batch policy learning under multiple constraints, and offer a systematic solution. We first propose a flexible meta-algorithm that admits any batch reinforcement learning and online learning procedure as subroutines. We then present a specific algorithmic instantiation and provide performance guarantees for the main objective and all constraints. To certify constraint satisfaction, we propose a new and simple method for off-policy policy evaluation (OPE) and derive PAC-style bounds. Our algorithm achieves strong empirical results in different domains, including in a challenging problem of simulated car driving subject to multiple constraints such as lane keeping and smooth driving. We also show experimentally that our OPE method outperforms other popular OPE techniques on a standalone basis, especially in a high-dimensional setting.

Key facts. Cited by 242 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 6; 2021: 40; 2022: 37; 2023: 45; 2024: 45; 2025: 38; 2026: 31.

242 papers.

Sort:

By area:

  1. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  2. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  3. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  4. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  5. 2022Tutorial on amortized optimizationcited by 83
  6. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  7. 2020Provably Good Batch Reinforcement Learning Without Great Explorationcited by 77
  8. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  9. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  10. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  11. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  12. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 63
38 more
  1. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  2. 2025WorldGym: World Model as An Environment for Policy Evaluationcited by 61
  3. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  4. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reductioncited by 56
  5. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 55
  6. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  7. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 45
  8. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
  9. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
  10. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimizationcited by 41
  11. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  12. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  13. 2021How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Reviewcited by 36
  14. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  15. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  16. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  17. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 31
  18. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  19. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  20. 2021Model-Based Offline Planning with Trajectory Pruningcited by 30
  21. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
  22. 2021A Simple Reward-free Approach to Constrained Reinforcement Learningcited by 28
  23. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  24. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  25. 2021Continuous Doubly Constrained Batch Reinforcement Learningcited by 27
  26. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  27. 2021Active Offline Policy Selectioncited by 26
  28. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26
  29. 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficientcited by 24
  30. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
  31. 2021Batched Neural Banditscited by 22
  32. 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
  33. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPscited by 22
  34. 2022Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systemscited by 22
  35. 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximationcited by 22
  36. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  37. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  38. 2023Survival Instinct in Offline Reinforcement Learningcited by 20

50 most cited of 242 All 242 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
  2. 2020BRPO: Batch Residual Policy Optimization
  3. 2026Behavior-Constrained Reinforcement Learning with Receding-Horizon Credit Assignment for High-Performance Control ↗
  4. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
  5. 2018Off-Policy Deep Reinforcement Learning without Exploration
  6. 2025Policy Constraint by Only Support Constraint for Offline Reinforcement Learning
  7. 2026Automatic Constraint Policy Optimization based on Continuous Constraint Interpolation Framework for Offline Reinforcement Learning
  8. 2021Offline RL Without Off-Policy Evaluation