mlsanityData to 7 Oct 2026

Density Constrained Reinforcement Learning

arXiv 2106.12764 ↗First posted 24 Jun 2021cs.LG cs.SY eess.SYcited by 25

Not on the map. Why?

Abstract (arXiv)

We study constrained reinforcement learning (CRL) from a novel perspective by setting constraints directly on state density functions, rather than the value functions considered by previous works. State density has a clear physical and mathematical interpretation, and is able to express a wide variety of constraints such as resource limits and safety requirements. Density constraints can also avoid the time-consuming process of designing and tuning cost functions required by value function-based constraints to encode system specifications. We leverage the duality between density functions and Q functions to develop an effective algorithm to solve the density constrained RL problem optimally and the constrains are guaranteed to be satisfied. We prove that the proposed algorithm converges to a near-optimal solution with a bounded error even when the policy update is imperfect. We use a set of comprehensive experiments to demonstrate the advantages of our approach over state-of-the-art CRL methods, with a wide range of density constrained tasks as well as standard CRL benchmarks such as Safety-Gym.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 21 of its 54 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

21 of 54 references.

References · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2017Constrained Policy Optimizationcited by 928
  5. 2018Reward Constrained Policy Optimizationcited by 397
  6. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  7. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  8. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2018Safe Exploration in Continuous Action Spacescited by 237
  11. 2020Projection-Based Constrained Policy Optimizationcited by 195
  12. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 144
9 more
  1. 2017A Tutorial on Kernel Density Estimation and Recent Advancescited by 100
  2. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  3. 2018Safe Reinforcement Learning with Model Uncertainty Estimatescited by 70
  4. 2017OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real Worldcited by 64
  5. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  6. 2017Boosting the Actor with Dual Criticcited by 14
  7. 2020Qualitative Controller Synthesis for Consumption Markov Decision Processescited by 9
  8. 2019Towards a Reinforcement Learning Environment Toolbox for Intelligent Electric Motor Controlcited by 6
  9. 2019Duality between density function and value function with applications in constrained optimal control and Markov Decision Processcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 7; 2024: 6; 2025: 3; 2026: 1.

25 papers.

Sort:

By area:

  1. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  2. 2022Reachability Constrained Reinforcement Learningcited by 52
  3. 2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 33
  4. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 29
  5. 2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 28
  6. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
  7. 2022SABLAS: Learning Safe Control for Black-box Dynamical Systemscited by 17
  8. 2022Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificatecited by 16
  9. 2024Global Reinforcement Learning: Beyond Linear and Convex Rewards via Submodular Semi-gradient Methodscited by 15
  10. 2023A Coupled Flow Approach to Imitation Learningcited by 12
  11. 2023Neural Operators for PDE Backstepping Control of First-Order Hyperbolic PIDE with Recycle and Delaycited by 11
  12. 2024Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learningcited by 11
13 more
  1. 2021Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updatescited by 5
  2. 2023Reinforcement Learning by Guided Safe Explorationcited by 5
  3. 2024Federated Reinforcement Learning with Constraint Heterogeneitycited by 2
  4. 2021Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific researchcited by 1
  5. 2023Learning Density-Based Correlated Equilibria for Markov Gamescited by 1
  6. 2024Efficient Multi-Task Reinforcement Learning via Task-Specific Action Correctioncited by 1
  7. 2024A Dual Perspective of Reinforcement Learning for Imposing Policy Constraintscited by 1
  8. 2023Safe Reinforcement Learning in Tensor Reproducing Kernel Hilbert Spacecited by 0
  9. 2024Physics-informed RL for Maximal Safety Probability Estimationcited by 0
  10. 2025DIAL: Distribution-Informed Adaptive Learning of Multi-Task Constraints for Safety-Critical Systemscited by 0
  11. 2025Ensemble RL through Classifier Models: Enhancing Risk-Return Trade-offs in Trading Strategiescited by 0
  12. 2025Situational-Constrained Sequential Resources Allocation via Reinforcement Learningcited by 0
  13. 2026Lyapunov-Guided Self-Alignment: Test-Time Adaptation for Offline Safe Reinforcement Learningcited by 0