mlsanityData to 14 Oct 2026

Redistribution-based Cost Inference Improves Sparse Safe Offline RL

arXiv 2608.12306 ↗First posted 12 Aug 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Safe offline RL typically assumes access to dense per-step cost annotations, but in practice supervisors provide only trajectory-level stop-feedback: a binary signal at the first unsafe transition, with no per-step attribution. We frame this as a temporal credit assignment problem and propose the Redistribution-based Cost Inference (RCI) framework, which converts sparse stop-feedback into dense per-step costs via return decomposition, then trains a constrained offline policy on the augmented dataset. We show that return-equivalent redistribution preserves the feasible policy set and the optimal Lagrangian in a CMDP, establishing that the transformation is lossless in theory while yielding better-conditioned cost critic learning in practice. Experiments on highway driving and robotic manipulation demonstrate substantially lower violation rates than sparse and classifier-based baselines, with robustness to heterogeneous dataset compositions and label noise.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 13 of its 23 references that are in the map. First posted on arXiv 12 Aug 2026.

Builds on

13 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning15 Jun
  2. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul

Other references · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2017Constrained Policy Optimizationcited by 983
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  7. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  8. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  9. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
  10. 2023Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approachcited by 19
  11. 2024Safety through feedback in Constrained RLcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline RL Policies Should be Trained to be Adaptive
  2. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  3. 2022When does return-conditioned supervised learning work for offline reinforcement learning?
  4. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation
  5. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
  6. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation
  7. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
  8. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraints