# Safe Reinforcement Learning with Receding-Horizon Safety Critic and Differentiable Action Shields

arXiv: https://arxiv.org/abs/2610.20255 · HTML: https://mlsanity.com/paper/2610.20255/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 30 of its 33 references that are in the map.
- First posted on arXiv 14 Oct 2026.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 30 of 30)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Control Barrier Function Based Quadratic Programs for Safety Critical Systems](https://mlsanity.com/paper/1609.06408/) (2016)
- [A predictive safety filter for learning-based control of constrained nonlinear dynamical systems](https://mlsanity.com/paper/1812.05506/) (2018)
- [Learning Control Barrier Functions from Expert Demonstrations](https://mlsanity.com/paper/2004.03315/) (2020)
- [Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion](https://mlsanity.com/paper/2401.17583/) (2024)
- [Learning for Safety-Critical Control with Control Barrier Functions](https://mlsanity.com/paper/1912.10099/) (2019)
- [Safe Reinforcement Learning Using Robust Control Barrier Functions](https://mlsanity.com/paper/2110.05415/) (2021)
- [How to Train Your Neural Control Barrier Function: Learning Safety Filters for Complex Input-Constrained Systems](https://mlsanity.com/paper/2310.15478/) (2023)
- [Safe Control Under Input Limits with Neural Control Barrier Functions](https://mlsanity.com/paper/2211.11056/) (2022)
- [Multi-Layered Safety for Legged Robots via Control Barrier Functions and Model Predictive Control](https://mlsanity.com/paper/2011.00032/) (2020)
- [Learning Barrier Functions with Memory for Robust Safe Navigation](https://mlsanity.com/paper/2011.01899/) (2020)
- [Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents](https://mlsanity.com/paper/2410.11671/) (2024)
- [CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions](https://mlsanity.com/paper/2510.14959/) (2025)
- [Safe Reinforcement Learning on the Constraint Manifold: Theory and Applications](https://mlsanity.com/paper/2404.09080/) (2024)
- [Predictive control barrier functions: Enhanced safety mechanisms for learning-based control](https://mlsanity.com/paper/2105.10241/) (2021)
- [Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions](https://mlsanity.com/paper/2305.09793/) (2023)
- [BarrierNet: A Safety-Guaranteed Layer for Neural Networks](https://mlsanity.com/paper/2111.11277/) (2021)
- [SafeFall: Learning Protective Control for Humanoid Robots](https://mlsanity.com/paper/2511.18509/) (2025)
- [Reinforcement Learning-based Receding Horizon Control using Adaptive Control Barrier Functions for Safety-Critical Systems](https://mlsanity.com/paper/2403.17338/) (2024)
- [SHIELD: Safety on Humanoids via CBFs In Expectation on Learned Dynamics](https://mlsanity.com/paper/2505.11494/) (2025)
- [Value Functions are Control Barrier Functions: Verification of Safe Policies using Control Theory](https://mlsanity.com/paper/2306.04026/) (2023)
- [Learning Conservative Neural Control Barrier Functions from Offline Data](https://mlsanity.com/paper/2505.00908/) (2025)
- [Neural Gaits: Learning Bipedal Locomotion via Control Barrier Functions and Zero Dynamics Policies](https://mlsanity.com/paper/2204.08120/) (2022)
- [Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learning](https://mlsanity.com/paper/2409.12045/) (2024)
- [Certificated Actor-Critic: Hierarchical Reinforcement Learning with Control Barrier Functions for Safe Navigation](https://mlsanity.com/paper/2501.17424/) (2025)
- [T-CBF: Traversability-based Control Barrier Function to Navigate Vertically Challenging Terrain](https://mlsanity.com/paper/2503.06083/) (2025)
- [Neural Control Barrier Functions from Physics Informed Neural Networks](https://mlsanity.com/paper/2504.11045/) (2025)
- [Learning Neural Control Barrier Functions from Expert Demonstrations using Inverse Constraint Learning](https://mlsanity.com/paper/2510.21560/) (2025)

## Abstract (arXiv)

Despite the Markov assumption, safety failures in robot reinforcement learning rarely occur instantaneously; they typically emerge over several steps as the policy enters unfamiliar states under disturbances and domain shift. We address this limitation by learning a receding-horizon safety critic that predicts whether a state will remain non-terminating over the next receding horizon, enabling anticipatory detection of impending locomotion failures. The critic is trained from rollout-derived horizon labels with a composite objective that combines horizon classification, control-barrier-function-inspired temporal consistency, and smoothness regularization, and we aggregate an ensemble of critics with a conservative Conditional Value at Risk estimate to account for uncertainty. Building on the learned critic, we derive a closed-form, differentiable CBF-style safety filter that minimally modifies the policy action to improve predicted safety, and integrate it into PPO with an anticipation loss to reduce policy-filter mismatch during training. In simulated quadruped locomotion, our method improves robustness under action noise, external perturbations, and terrain shifts, reducing catastrophic failures while maintaining competitive returns.