mlsanityData to 7 Oct 2026

State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewards

arXiv 2102.11941 ↗First posted 23 Feb 2021cs.LG cs.RO math.OCcited by 37

Not on the map. Why?

Abstract (arXiv)

A common formulation of constrained reinforcement learning involves multiple rewards that must individually accumulate to given thresholds. In this class of problems, we show a simple example in which the desired optimal policy cannot be induced by any weighted linear combination of rewards. Hence, there exist constrained reinforcement learning problems for which neither regularized nor classical primal-dual methods yield optimal policies. This work addresses this shortcoming by augmenting the state with Lagrange multipliers and reinterpreting primal-dual methods as the portion of the dynamics that drives the multipliers evolution. This approach provides a systematic state augmentation procedure that is guaranteed to solve reinforcement learning problems with constraints. Thus, as we illustrate by an example, while previous methods can fail at finding optimal policies, running the dual dynamics while executing the augmented policy yields an algorithm that provably samples actions from the optimal policy.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 9 of its 41 references that are in the map. First posted on arXiv 23 Feb 2021.

Builds on

9 of 41 references.

References · 9

  1. 2017Constrained Policy Optimizationcited by 928
  2. 2018Reward Constrained Policy Optimizationcited by 397
  3. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  4. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  5. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  6. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 144
  7. 2018Simple random search provides a competitive approach to reinforcement learningcited by 75
  8. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 75
  9. 2013Variance Adjusted Actor Critic Algorithmscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 7; 2024: 5; 2025: 9; 2026: 10.

37 papers.

Sort:

By area:

  1. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  2. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
  3. 2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 27
  4. 2021Safe Pontryagin Differentiable Programmingcited by 25
  5. 2024Truly No-Regret Learning in Constrained MDPscited by 23
  6. 2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 19
  7. 2022State-Augmented Learnable Algorithms for Resource Management in Wireless Networkscited by 13
  8. 2023Probabilistic Constraint for Safety-Critical Reinforcement Learningcited by 11
  9. 2023A safe exploration approach to constrained Markov decision processescited by 10
  10. 2025Intersectional Fairness in Reinforcement Learning with Large State and Constraint Spacescited by 5
  11. 2025Multi-Objective Reinforcement Learning with Max-Min Criterion: A Game-Theoretic Approachcited by 5
  12. 2023ACPO: A Policy Optimization Algorithm for Average MDPs with Constraintscited by 4
25 more
  1. 2024Adaptive Primal-Dual Method for Safe Reinforcement Learningcited by 4
  2. 2022Constrained Reinforcement Learning via Dissipative Saddle Flow Dynamicscited by 3
  3. 2024Multi-agent assignment via state augmented reinforcement learningcited by 3
  4. 2025Opportunistic Routing in Wireless Communications via Learnable State-Augmented Policiescited by 3
  5. 2025Fast State-Augmented Learning for Wireless Resource Allocation with Dual Variable Regressioncited by 3
  6. 2024Learning to Slice Wi-Fi Networks: A State-Augmented Primal-Dual Approachcited by 2
  7. 2025Constrained Sliced Wasserstein Embeddingcited by 2
  8. 2023Learning State-Augmented Policies for Information Routing in Communication Networkscited by 1
  9. 2023Resilient Constrained Reinforcement Learningcited by 1
  10. 2026Scalable Constrained Multi-Agent Reinforcement Learning via State Augmentation and Consensus for Separable Dynamicscited by 1
  11. 2022A State-Augmented Approach for Learning Optimal Resource Management Decisions in Wireless Networkscited by 0
  12. 2023SCPO: Safe Reinforcement Learning with Safety Critic Policy Optimizationcited by 0
  13. 2024Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPscited by 0
  14. 2025Navigating Demand Uncertainty in Container Shipping: Deep Reinforcement Learning for Enabling Adaptive and Feasible Master Stowage Planningcited by 0
  15. 2025ProSh: Probabilistic Shielding for Model-free Reinforcement Learningcited by 0
  16. 2025State-Augmented Graphs for Circular Economy Triagecited by 0
  17. 2026Multi-Objective Reinforcement Learning for Large-Scale Tote Allocation in Human-Robot Collaborative Fulfillment Centerscited by 0
  18. 2026Price of Fairness in Short-Term and Long-Term Algorithmic Selectionscited by 0
  19. 2026Constrained Multi-Objective Reinforcement Learning with Max-Min Criterioncited by 0
  20. 2026Coordination Graphs for Constrained Multi-Agent Reinforcement Learningcited by 0
  21. 2026Limit Analysis of Graph Neural Networks with Wireless Conflict Graphscited by 0
  22. 2026Feasible Flow Matching for Graph Reconstruction via Within-Sampling Primal-Dual Guidancecited by 0
  23. 2026Learning to Harvest Without Collapse in a Regenerative Commons: A Lagrangian Frameworkcited by 0
  24. 2026Learning Infinite-Horizon Average-Reward CMDPs via State Augmentationcited by 0
  25. 2026The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learningcited by 0