# Recursive Constraints to Prevent Instability in Constrained Reinforcement Learning

arXiv: https://arxiv.org/abs/2201.07958 · HTML: https://mlsanity.com/paper/2201.07958/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 11 references that are in the map.
- First posted on arXiv 20 Jan 2022.

## Builds on (most cited first, 1 of 1)

- [Deep Constrained Q-learning](https://mlsanity.com/paper/2003.09398/) (2020)

## Built on by (most cited first, 2 of 2)

- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Recursively-Constrained Partially Observable Markov Decision Processes](https://mlsanity.com/paper/2310.09688/) (2023)

## Abstract (arXiv)

We consider the challenge of finding a deterministic policy for a Markov decision process that uniformly (in all states) maximizes one reward subject to a probabilistic constraint over a different reward. Existing solutions do not fully address our precise problem definition, which nevertheless arises naturally in the context of safety-critical robotic systems. This class of problem is known to be hard, but the combined requirements of determinism and uniform optimality can create learning instability. In this work, after describing and motivating our problem with a simple example, we present a suitable constrained reinforcement learning algorithm that prevents learning instability, using recursive constraints. Our proposed approach admits an approximative form that improves efficiency and is conservative w.r.t. the constraint.