mlsanityData to 14 Oct 2026

Direct Behavior Specification via Constrained Reinforcement Learning

arXiv 2112.12228 ↗First posted 22 Dec 2021cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

The standard formulation of Reinforcement Learning lacks a practical way of specifying what are admissible and forbidden behaviors. Most often, practitioners go about the task of behavior specification by manually engineering the reward function, a counter-intuitive process that requires several iterations and is prone to reward hacking by the agent. In this work, we argue that constrained RL, which has almost exclusively been used for safe RL, also has the potential to significantly reduce the amount of work spent for reward specification in applied RL projects. To this end, we propose to specify behavioral preferences in the CMDP framework and to use Lagrangian methods to automatically weigh each of these behavioral constraints. Specifically, we investigate how CMDPs can be adapted to solve goal-based tasks while adhering to several constraints simultaneously. We evaluate this framework on a set of continuous control tasks relevant to the application of Reinforcement Learning for NPC design in video games.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 36 of its 75 references that are in the map. First posted on arXiv 22 Dec 2021.

Builds on

36 of 75 references.

References · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2016Concrete Problems in AI Safetycited by 1,642
  7. 2016Generative Adversarial Imitation Learningcited by 1,597
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  9. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  10. 2017Constrained Policy Optimizationcited by 983
  11. 2018Learning Dexterous In-Hand Manipulationcited by 687
  12. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
24 more
  1. 2018Reward Constrained Policy Optimizationcited by 417
  2. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  3. 2019Challenges of Real-World Reinforcement Learningcited by 291
  4. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 270
  5. 2017Inverse Reward Designcited by 249
  6. 2018Safe Exploration in Continuous Action Spacescited by 247
  7. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 204
  8. 2020Projection-Based Constrained Policy Optimizationcited by 203
  9. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  10. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  11. 2020Conservative Safety Critics for Explorationcited by 113
  12. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 103
  13. 2020Safe Reinforcement Learning via Curriculum Inductioncited by 74
  14. 2019A Survey of Deep Reinforcement Learning in Video Gamescited by 73
  15. 2021On the Expressivity of Markov Rewardcited by 71
  16. 2021Augmenting Automated Game Testing with Deep Reinforcement Learningcited by 38
  17. 2021Adversarial Reinforcement Learning for Procedural Content Generationcited by 36
  18. 2021Improving Playtesting Coverage via Curiosity Driven Reinforcement Learning Agentscited by 28
  19. 2020Deep Reinforcement Learning for Navigation in AAA Video Gamescited by 27
  20. 2020Balancing Constraints and Rewards with Meta-Gradient D4PGcited by 18
  21. 2020"It's Unwieldy and It Takes a Lot of Time." Challenges and Opportunities for Creating Agents in Commercial Gamescited by 17
  22. 2021Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigationcited by 17
  23. 2018Simplifying Reward Design through Divide-and-Conquercited by 12
  24. 2018Active Inverse Reward Designcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 6; 2024: 12; 2025: 3; 2026: 3.

31 papers.

Sort:

By area:

  1. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  2. 2022Verifying Learning-Based Robotic Navigation Systemscited by 33
  3. 2023Goal-conditioned GFlowNets for Controllable Multi-Objective Molecular Designcited by 17
  4. 2022Towards Informed Design and Validation Assistance in Computer Games Using Imitation Learningcited by 13
  5. 2024Verification-Guided Shielding for Deep Reinforcement Learningcited by 13
  6. 2024Constrain Alignment with Sparse Autoencoderscited by 12
  7. 2022Constrained Reinforcement Learning for Robotics via Scenario-Based Programmingcited by 11
  8. 2022Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representationcited by 11
  9. 2022Automated Play-Testing Through RL Based Human-Like Play-Styles Generationcited by 11
  10. 2023Safe Deep Reinforcement Learning by Verifying Task-Level Propertiescited by 9
  11. 2023Decision-Making Under Uncertainty: Beyond Probabilitiescited by 9
  12. 2023Online Safety Property Collection and Refinement for Safe Deep Reinforcement Learning in Mapless Navigationcited by 7
19 more
  1. 2024Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolutioncited by 7
  2. 2025HASARD: A Benchmark for Vision-Based Safe Reinforcement Learning in Embodied Agentscited by 6
  3. 2024Improving Policy Optimization via varepsilon-Retraincited by 5
  4. 2025Position: Adopt Constraints Over Fixed Penalties in Deep Learningcited by 5
  5. 2024Long and Short-Term Constraints Driven Safe Reinforcement Learning for Autonomous Drivingcited by 4
  6. 2023Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidancecited by 3
  7. 2024Safe Multiagent Coordination via Entropic Explorationcited by 3
  8. 2025Central Path Proximal Policy Optimizationcited by 3
  9. 2024Do No Harm: A Counterfactual Approach to Safe Reinforcement Learningcited by 2
  10. 2026Constrained Group Relative Policy Optimizationcited by 2
  11. 2026CRAX: Fast Safe Reinforcement Learning Benchmarkingcited by 2
  12. 2023Resilient Constrained Reinforcement Learningcited by 1
  13. 2022On the importance of data collection for training general goal-reaching policiescited by 0
  14. 2022Tiered Reward: Designing Rewards for Specification and Fast Learning of Desired Behaviorcited by 0
  15. 2024Adaptive Autopilot: Constrained DRL for Diverse Driving Behaviorscited by 0
  16. 2024Adversarial Constrained Policy Optimization: Improving Constrained Reinforcement Learning by Adapting Budgetscited by 0
  17. 2024Building Altruistic and Moral AI Agent with Brain-inspired Emotional Empathy Mechanismscited by 0
  18. 2024Effective Reward Specification in Deep Reinforcement Learningcited by 0
  19. 2026SCoCaT: Success Conditioned Constrained Reinforcement Learning for Spacecraft Dockingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey of Constraint Formulations in Safe Reinforcement Learning
  2. 2025RLAF: Reinforcement Learning from Automaton Feedback
  3. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Tasks
  4. 2024A Dual Perspective of Reinforcement Learning for Imposing Policy Constraints
  5. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithms
  6. 2025Reinforcement Learning based Constrained Optimal Control: an Interpretable Reward Design
  7. 2021Continuous Doubly Constrained Batch Reinforcement Learning
  8. 2023Behavior Alignment via Reward Function Optimization