mlsanityData to 14 Oct 2026

TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labels

arXiv 2504.12557 ↗First posted 17 Apr 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Ensuring safe behavior in reinforcement learning (RL) is challenging when safety constraints are implicit and cannot be densely measured. In many settings, supervision is limited to coarse approvals or rejections of whole trajectories (e.g., whether a rollout remained within an unknown safety threshold). We propose TraCeS (Trajectory-based Constraint Estimation for Safety), a method for learning per-timestep violation credit from such sparse trajectory-level labels. TraCeS trains a sequential violation estimator whose per-step credits factorize the predicted probability that a trajectory has not yet violated the constraint, and integrates this learned signal into constrained policy optimization. The method requires neither a known cost function nor a known threshold, and remains compatible with standard continuous-control algorithms. We provide a theoretical analysis of the approximation gap introduced by the learning objective, and demonstrate empirically that TraCeS improves constraint satisfaction and feedback efficiency over baselines across multiple continuous-control benchmarks, including long-horizon tasks and settings with noisy or inconsistent labels.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 29 of its 50 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

29 of 50 references.

References · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2017Constrained Policy Optimizationcited by 983
  5. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  6. 2017Safe Reinforcement Learning via Shieldingcited by 449
  7. 2018Reward Constrained Policy Optimizationcited by 417
  8. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  9. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  10. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  11. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  12. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
17 more
  1. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  2. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  3. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 111
  4. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  5. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  6. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
  7. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  8. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 59
  9. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 58
  10. 2020Inverse Constrained Reinforcement Learningcited by 46
  11. 2018Penalizing side effects using stepwise relative reachabilitycited by 38
  12. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  13. 2019Maximum Likelihood Constraint Inference for Inverse Reinforcement Learningcited by 34
  14. 2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
  15. 2022Benchmarking Constraint Inference in Inverse Reinforcement Learningcited by 18
  16. 2023Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learningcited by 12
  17. 2024Safety through feedback in Constrained RLcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
  2. 2024Revisiting Safe Exploration in Safe Reinforcement learning
  3. 2026Decoupled Guidance Diffusion for Adaptive Offline Safe Reinforcement Learning
  4. 2025Safety Representations for Safer Policy Learning
  5. 2026From Cumulative Constraints to Adaptive Runtime Safety Control for Nonstationary Reinforcement Learning
  6. 2026Evaluation Metrics for Safe Reinforcement Learning
  7. 2024A Survey of Constraint Formulations in Safe Reinforcement Learning
  8. 2026Scenario Generation for Risk-Aware Reinforcement Learning with Probably Approximately Safe Guarantees