mlsanityData to 14 Oct 2026

Certified Safety Curation: Distribution-Free Guarantees for Safe Offline Reinforcement Learning

arXiv 2609.12014 ↗First posted 10 Sep 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Safe offline reinforcement learning assumes a cost function on every transition. We ask what remains possible when safety can be judged only by comparing short clips and occasionally asking whether an episode exceeded its budget. Certified safety curation answers with a filter-then-clone pipeline: a state-only value trained from segment comparisons scores whole trajectories, Learn-then-Test calibration certifies a selection threshold under a distribution-free (α, δ) bound on the unsafe fraction of the selection, and behavior cloning follows. What is certified is the training set, not the policy, whose cost we report rather than bound. Where no threshold attains the target, the procedure refuses. We are not aware of prior work certifying the composition of a training set for offline RL or imitation. Oracle controls justify the design: reweighting individual transitions fails even with an exact value, so the value selects whole trajectories. The policies satisfy the cost budget on twelve of fifteen DSRL tasks, matching a clone of the ground-truth safe subset, which needs a label on every trajectory. Retrained on the certified selection, the strongest full-label method becomes safe where no setting of its own cost target rescues it. Refusal is predictable: the certificate's probability has a closed form in the purity of the pool's top quantile, which the calibration sample estimates and through which the scorer enters.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 30 references that are in the map. First posted on arXiv 10 Sep 2026.

Builds on

26 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment20 May
  2. 2026OSIL: Learning Offline Safe Imitation Policies with Safety Inferred from Non-preferred Trajectories11 Feb
  3. 2024Offline Safe Reinforcement Learning Using Trajectory Classification19 Dec
  4. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning15 Jun
  5. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning14 Feb
  6. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul

Other references · 20

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  4. 2022Conformal Risk Controlcited by 299
  5. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 288
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  7. 2020Critic Regularized Regressioncited by 275
  8. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
  9. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  10. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  11. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  12. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
8 more
  1. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  2. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  3. 2022Selection by Prediction with Conformal p-valuescited by 69
  4. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 60
  5. 2022Conformal Off-Policy Prediction in Contextual Banditscited by 25
  6. 2025Learning Robot Safety from Sparse Human Feedback using Conformal Predictioncited by 2
  7. 2025Offline RLAIF: Piloting VLM Feedback for RL via SFOcited by 2
  8. 2025Offline Safe Policy Optimization From Heterogeneous Feedbackcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Safe-RULE: Safe Reinforcement UnLEarning
  2. 2025Safe Reinforcement Learning with Minimal Supervision
  3. 2026Evaluation Metrics for Safe Reinforcement Learning
  4. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
  5. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
  6. 2026Conditional Sequence Modeling for Safe Reinforcement Learning
  7. 2026LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy
  8. 2024Revisiting Safe Exploration in Safe Reinforcement learning