Data to 7 Oct 2026

Reward Constrained Policy Optimization

arXiv 1805.11074 ↗First posted 28 May 2018cs.LG cs.AI stat.MLcited by 397

Not on the map. Why?

Cited by 27 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work we present a novel multi-timescale approach for constrained policy optimization, called `Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.

Key facts. Cited by 397 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 51; 2022: 54; 2023: 72; 2024: 79; 2025: 55; 2026: 86.

397 papers.

Sort:

By area:

2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
2021Reward is enough for convex MDPscited by 82
2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 78
2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65

38 more

2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 60
2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 55
2024Agile But Safe: Learning Collision-Free High-Speed Legged Locomotioncited by 53
2022Reachability Constrained Reinforcement Learningcited by 52
2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 47
2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 46
2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 45
2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
2023Safe Reinforcement Learning via Probabilistic Logic Shieldscited by 41
2021Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challengescited by 40
2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
2021State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewardscited by 37
2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
2022Near-Optimal Sample Complexity Bounds for Constrained MDPscited by 36
2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 35
2021DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learningcited by 34
2021Safe Reinforcement Learning with Linear Function Approximationcited by 33
2023Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotioncited by 33
2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 33
2022Aligning to Social Norms and Values in Interactive Narrativescited by 32
2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
2021Model-based Constrained Reinforcement Learning using Generalized Control Barrier Functioncited by 30
2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 30
2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
2022Learning Infinite-Horizon Average-Reward Markov Decision Processes with Constraintscited by 29
2022Safe Reinforcement Learning for Legged Locomotioncited by 29
2021A Simple Reward-free Approach to Constrained Reinforcement Learningcited by 28
2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 28
2022MuZero with Self-competition for Rate Control in VP9 Video Compressioncited by 27
2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 27
2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 27
2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 26
2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26
2021Density Constrained Reinforcement Learningcited by 25
2021Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learningcited by 25
2024Truly No-Regret Learning in Constrained MDPscited by 23

50 most cited of 397 All 397 (JSON)