mlsanityData to 7 Oct 2026

Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs

arXiv 2106.00099 ↗First posted 31 May 2021cs.LGcited by 22

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

We study the problem of Safe Policy Improvement (SPI) under constraints in the offline Reinforcement Learning (RL) setting. We consider the scenario where: (i) we have a dataset collected under a known baseline policy, (ii) multiple reward signals are received from the environment inducing as many objectives to optimize. We present an SPI formulation for this RL setting that takes into account the preferences of the algorithm's user for handling the trade-offs for different reward signals while ensuring that the new policy performs at least as well as the baseline policy along each individual objective. We build on traditional SPI algorithms and propose a novel method based on Safe Policy Iteration with Baseline Bootstrapping (SPIBB, Laroche et al., 2019) that provides high probability guarantees on the performance of the agent in the true environment. We show the effectiveness of our method on a synthetic grid-world safety task as well as in a real-world critical care context to learn a policy for the administration of IV fluids and vasopressors to treat sepsis.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 20 of its 74 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

20 of 74 references.

Closest ancestors · 2

Same direction, newest first

  1. 2019Safe Policy Improvement with an Estimated Baseline Policy11 Sep
  2. 2019Safe Policy Improvement with Soft Baseline Bootstrapping11 Jul

Other references · 18

  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 633
  4. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  5. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  6. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  7. 2019Challenges of Real-World Reinforcement Learningcited by 282
  8. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  9. 2019Batch Policy Learning under Constraintscited by 242
  10. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
  11. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 166
  12. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
6 more
  1. 2017AI Safety Gridworldscited by 108
  2. 2017Environment-Independent Task Specifications via GLTLcited by 67
  3. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
  4. 2020A Distributional View on Multi-Objective Policy Optimizationcited by 51
  5. 2020Clinician-in-the-Loop Decision Making: Reinforcement Learning with Near-Optimal Set-Valued Policiescited by 24
  6. 2020Trajectory Inspection: A Method for Iterative Clinician-Driven Design of Reinforcement Learning Studiescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 4; 2025: 5; 2026: 2.

22 papers.

Sort:

By area:

  1. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
  2. 2023Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithmscited by 33
  3. 2022Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Stochastic Approachcited by 14
  4. 2022Behavior Prior Representation learning for Offline Reinforcement Learningcited by 12
  5. 2024A CMDP-within-online framework for Meta-Safe Reinforcement Learningcited by 12
  6. 2023Safe Policy Improvement for POMDPs via Finite-State Controllerscited by 10
  7. 2023More for Less: Safe Policy Improvement With Stronger Performance Guaranteescited by 9
  8. 2024Policy-regularized Offline Multi-objective Reinforcement Learningcited by 9
  9. 2022Offline Policy Optimization with Eligible Actionscited by 7
  10. 2024MGDA Converges under Generalized Smoothness, Provablycited by 6
  11. 2024MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learningcited by 4
  12. 2025Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatmentcited by 4
10 more
  1. 2025Multiple Wasserstein Gradient Descent Algorithm for Multi-Objective Distributional Optimizationcited by 3
  2. 2023Robust Lagrangian and Adversarial Policy Gradient for Robust Constrained Markov Decision Processescited by 2
  3. 2023Pessimistic Off-Policy Multi-Objective Optimizationcited by 2
  4. 2025A Provable Approach for End-to-End Safe Reinforcement Learningcited by 2
  5. 2025Data-Efficient Safe Policy Improvement Using Parametric Structurecited by 2
  6. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learningcited by 2
  7. 2025Seldonian Reinforcement Learning for Ad Hoc Teamworkcited by 1
  8. 2026The hidden risks of temporal resampling in clinical reinforcement learningcited by 1
  9. 2022Non-Markovian policies occupancy measurescited by 0
  10. 2023Adaptive Aggregation for Safety-Critical Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Safe Policy Improvement Approaches and their Limitations
  2. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  3. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning
  4. 2025Deep SPI: Safe Policy Improvement via World Models
  5. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
  6. 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning
  7. 2025Online Optimization for Offline Safe Reinforcement Learning
  8. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning