# Resilient Constrained Reinforcement Learning

arXiv: https://arxiv.org/abs/2312.17194 · HTML: https://mlsanity.com/paper/2312.17194/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 61 references that are in the map.
- First posted on arXiv 28 Dec 2023.

## Builds on (most cited first, 24 of 24)

- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Constrained Reinforcement Learning Has Zero Duality Gap](https://mlsanity.com/paper/1910.13393/) (2019)
- [Exploration-Exploitation in Constrained MDPs](https://mlsanity.com/paper/2003.02189/) (2020)
- [Provably Efficient Safe Exploration via Primal-Dual Policy Optimization](https://mlsanity.com/paper/2003.00534/) (2020)
- [On the Convergence Rates of Policy Gradient Methods](https://mlsanity.com/paper/2201.07443/) (2022)
- [Reward is enough for convex MDPs](https://mlsanity.com/paper/2106.00661/) (2021)
- [Safe Policies for Reinforcement Learning via Primal-Dual Methods](https://mlsanity.com/paper/1911.09101/) (2019)
- [Constrained Decision Transformer for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2302.07351/) (2023)
- [Reward Uncertainty for Exploration in Preference-based Reinforcement Learning](https://mlsanity.com/paper/2205.12401/) (2022)
- [On Linear Convergence of Policy Gradient Methods for Finite MDPs](https://mlsanity.com/paper/2007.11120/) (2020)
- [Sim2Real Transfer for Reinforcement Learning without Dynamics Randomization](https://mlsanity.com/paper/2002.11635/) (2020)
- [Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings](https://mlsanity.com/paper/2008.06622/) (2020)
- [State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewards](https://mlsanity.com/paper/2102.11941/) (2021)
- [Direct Behavior Specification via Constrained Reinforcement Learning](https://mlsanity.com/paper/2112.12228/) (2021)
- [Robot Navigation in Constrained Pedestrian Environments using Reinforcement Learning](https://mlsanity.com/paper/2010.08600/) (2020)
- [Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs](https://mlsanity.com/paper/2306.11700/) (2023)
- [Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPs](https://mlsanity.com/paper/2206.02346/) (2022)
- [Reinforcement Learning for Feedback-Enabled Cyber Resilience](https://mlsanity.com/paper/2107.00783/) (2021)
- [SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2301.12203/) (2023)
- [Resilient Constrained Learning](https://mlsanity.com/paper/2306.02426/) (2023)
- [Welfare Maximization Algorithm for Solving Budget-Constrained Multi-Component POMDPs](https://mlsanity.com/paper/2303.10302/) (2023)
- [Resilient Control: Compromising to Adapt](https://mlsanity.com/paper/2004.03726/) (2020)

## Built on by (most cited first, 1 of 1)

- [OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2407.14653/) (2024)

## Abstract (arXiv)

We study a class of constrained reinforcement learning (RL) problems in which multiple constraint specifications are not identified before training. It is challenging to identify appropriate constraint specifications due to the undefined trade-off between the reward maximization objective and the constraint satisfaction, which is ubiquitous in constrained decision-making. To tackle this issue, we propose a new constrained RL approach that searches for policy and constraint specifications together. This method features the adaptation of relaxing the constraint according to a relaxation cost introduced in the learning objective. Since this feature mimics how ecological systems adapt to disruptions by altering operation, our approach is termed as resilient constrained RL. Specifically, we provide a set of sufficient conditions that balance the constraint satisfaction and the reward maximization in notion of resilient equilibrium, propose a tractable formulation of resilient constrained policy optimization that takes this equilibrium as an optimal solution, and advocate two resilient constrained policy search algorithms with non-asymptotic convergence guarantees on the optimality gap and constraint satisfaction. Furthermore, we demonstrate the merits and the effectiveness of our approach in computational experiments.