# State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewards

arXiv: https://arxiv.org/abs/2102.11941 · HTML: https://mlsanity.com/paper/2102.11941/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 41 references that are in the map.
- First posted on arXiv 23 Feb 2021.

## Builds on (most cited first, 9 of 9)

- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift](https://mlsanity.com/paper/1908.00261/) (2019)
- [Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach](https://mlsanity.com/paper/1506.02188/) (2015)
- [Constrained Reinforcement Learning Has Zero Duality Gap](https://mlsanity.com/paper/1910.13393/) (2019)
- [Safe Policies for Reinforcement Learning via Primal-Dual Methods](https://mlsanity.com/paper/1911.09101/) (2019)
- [Simple random search provides a competitive approach to reinforcement learning](https://mlsanity.com/paper/1803.07055/) (2018)
- [Variance Adjusted Actor Critic Algorithms](https://mlsanity.com/paper/1310.3697/) (2013)

## Built on by (most cited first, 30 of 40)

- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentation](https://mlsanity.com/paper/2202.06558/) (2022)
- [Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs](https://mlsanity.com/paper/2306.11700/) (2023)
- [Safe Pontryagin Differentiable Programming](https://mlsanity.com/paper/2105.14937/) (2021)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [Truly No-Regret Learning in Constrained MDPs](https://mlsanity.com/paper/2402.15776/) (2024)
- [State-Augmented Learnable Algorithms for Resource Management in Wireless Networks](https://mlsanity.com/paper/2207.02242/) (2022)
- [Probabilistic Constraint for Safety-Critical Reinforcement Learning](https://mlsanity.com/paper/2306.17279/) (2023)
- [A Review of Safe Reinforcement Learning Methods for Modern Power Systems](https://mlsanity.com/paper/2407.00304/) (2024)
- [A safe exploration approach to constrained Markov decision processes](https://mlsanity.com/paper/2312.00561/) (2023)
- [Intersectional Fairness in Reinforcement Learning with Large State and Constraint Spaces](https://mlsanity.com/paper/2502.11828/) (2025)
- [Multi-Objective Reinforcement Learning with Max-Min Criterion: A Game-Theoretic Approach](https://mlsanity.com/paper/2510.20235/) (2025)
- [ACPO: A Policy Optimization Algorithm for Average MDPs with Constraints](https://mlsanity.com/paper/2302.00808/) (2023)
- [Adaptive Primal-Dual Method for Safe Reinforcement Learning](https://mlsanity.com/paper/2402.00355/) (2024)
- [Operator Splitting for Convex Constrained Markov Decision Processes](https://mlsanity.com/paper/2412.14002/) (2024)
- [Constrained Reinforcement Learning via Dissipative Saddle Flow Dynamics](https://mlsanity.com/paper/2212.01505/) (2022)
- [Multi-agent assignment via state augmented reinforcement learning](https://mlsanity.com/paper/2406.01782/) (2024)
- [Opportunistic Routing in Wireless Communications via Learnable State-Augmented Policies](https://mlsanity.com/paper/2503.03736/) (2025)
- [Fast State-Augmented Learning for Wireless Resource Allocation with Dual Variable Regression](https://mlsanity.com/paper/2506.18748/) (2025)
- [Learning to Slice Wi-Fi Networks: A State-Augmented Primal-Dual Approach](https://mlsanity.com/paper/2405.05748/) (2024)
- [Constrained Sliced Wasserstein Embedding](https://mlsanity.com/paper/2506.02203/) (2025)
- [Learning State-Augmented Policies for Information Routing in Communication Networks](https://mlsanity.com/paper/2310.00248/) (2023)
- [Resilient Constrained Reinforcement Learning](https://mlsanity.com/paper/2312.17194/) (2023)
- [Scalable Constrained Multi-Agent Reinforcement Learning via State Augmentation and Consensus for Separable Dynamics](https://mlsanity.com/paper/2605.30461/) (2026)
- [A State-Augmented Approach for Learning Optimal Resource Management Decisions in Wireless Networks](https://mlsanity.com/paper/2210.16412/) (2022)
- [SCPO: Safe Reinforcement Learning with Safety Critic Policy Optimization](https://mlsanity.com/paper/2311.00880/) (2023)
- [Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs](https://mlsanity.com/paper/2408.10015/) (2024)
- [Navigating Demand Uncertainty in Container Shipping: Deep Reinforcement Learning for Enabling Adaptive and Feasible Master Stowage Planning](https://mlsanity.com/paper/2502.12756/) (2025)
- [ProSh: Probabilistic Shielding for Model-free Reinforcement Learning](https://mlsanity.com/paper/2510.15720/) (2025)
- [QoS-Aware State-Augmented Learnable Framework for 5G NR-U/Wi-Fi Coexistence: Impact of Parameter Selection and Enhanced Collision Resolution](https://mlsanity.com/paper/2512.13393/) (2025)

## Abstract (arXiv)

A common formulation of constrained reinforcement learning involves multiple rewards that must individually accumulate to given thresholds. In this class of problems, we show a simple example in which the desired optimal policy cannot be induced by any weighted linear combination of rewards. Hence, there exist constrained reinforcement learning problems for which neither regularized nor classical primal-dual methods yield optimal policies. This work addresses this shortcoming by augmenting the state with Lagrange multipliers and reinterpreting primal-dual methods as the portion of the dynamics that drives the multipliers evolution. This approach provides a systematic state augmentation procedure that is guaranteed to solve reinforcement learning problems with constraints. Thus, as we illustrate by an example, while previous methods can fail at finding optimal policies, running the dual dynamics while executing the augmented policy yields an algorithm that provably samples actions from the optimal policy.