# Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraints

arXiv: https://arxiv.org/abs/2304.08743 · HTML: https://mlsanity.com/paper/2304.08743/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 18 Apr 2023.

## Builds on (most cited first, 20 of 20)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World](https://mlsanity.com/paper/1709.07643/) (2017)
- [Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization](https://mlsanity.com/paper/2105.08881/) (2021)
- [Resource Constrained Deep Reinforcement Learning](https://mlsanity.com/paper/1812.00600/) (2018)
- [Safe Optimal Control Using Stochastic Barrier Functions and Deep Forward-Backward SDEs](https://mlsanity.com/paper/2009.01196/) (2020)
- [Clipped Action Policy Gradient](https://mlsanity.com/paper/1802.07564/) (2018)
- [Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization](https://mlsanity.com/paper/2102.11055/) (2021)
- [Solving Online Threat Screening Games using Constrained Action Space Reinforcement Learning](https://mlsanity.com/paper/1911.08799/) (2019)
- [Sample-Specific Output Constraints for Neural Networks](https://mlsanity.com/paper/2003.10258/) (2020)

## Built on by (most cited first, 11 of 11)

- [Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs](https://mlsanity.com/paper/2503.12932/) (2025)
- [Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?](https://mlsanity.com/paper/2509.12833/) (2025)
- [DFWLayer: Differentiable Frank-Wolfe Optimization Layer](https://mlsanity.com/paper/2308.10806/) (2023)
- [Leveraging Constraint Violation Signals For Action-Constrained Reinforcement Learning](https://mlsanity.com/paper/2502.10431/) (2025)
- [Action-Constrained Imitation Learning](https://mlsanity.com/paper/2508.14379/) (2025)
- [Improving Stochastic Action-Constrained Reinforcement Learning via Truncated Distributions](https://mlsanity.com/paper/2511.22406/) (2025)
- [Adaptive Safe Reinforcement Learning-Enabled Optimization of Battery Fast-Charging Protocols](https://mlsanity.com/paper/2406.12309/) (2024)
- [Differentiable Policy Transport over Multi-Layer Network Feasibility Geometry](https://mlsanity.com/paper/2609.26068/) (2026)
- [Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints](https://mlsanity.com/paper/2405.03005/) (2024)
- [Navigating Demand Uncertainty in Container Shipping: Deep Reinforcement Learning for Enabling Adaptive and Feasible Master Stowage Planning](https://mlsanity.com/paper/2502.12756/) (2025)
- [Constraint-Enhanced Reinforcement Learning Based on Dynamic Decoupled Spherical Radial Squashing](https://mlsanity.com/paper/2605.04185/) (2026)

## Abstract (arXiv)

This study presents a benchmark for evaluating action-constrained reinforcement learning (RL) algorithms. In action-constrained RL, each action taken by the learning system must comply with certain constraints. These constraints are crucial for ensuring the feasibility and safety of actions in real-world systems. We evaluate existing algorithms and their novel variants across multiple robotics control environments, encompassing multiple action constraint types. Our evaluation provides the first in-depth perspective of the field, revealing surprising insights, including the effectiveness of a straightforward baseline approach. The benchmark problems and associated code utilized in our experiments are made available online at github.com/omron-sinicx/action-constrained-RL-benchmark for further research and development.