# Clipped Action Policy Gradient

arXiv: https://arxiv.org/abs/1802.07564 · HTML: https://mlsanity.com/paper/1802.07564/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2018.

## Built on by (most cited first, 19 of 19)

- [KnowGPT: Knowledge Graph based Prompting for Large Language Models](https://mlsanity.com/paper/2312.06185/) (2023)
- [Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization](https://mlsanity.com/paper/2102.11055/) (2021)
- [Pseudo-Labeled Auto-Curriculum Learning for Semi-Supervised Keypoint Localization](https://mlsanity.com/paper/2201.08613/) (2022)
- [Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraints](https://mlsanity.com/paper/2304.08743/) (2023)
- [Measuring Progress in Deep Reinforcement Learning Sample Efficiency](https://mlsanity.com/paper/2102.04881/) (2021)
- [Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPs](https://mlsanity.com/paper/2102.10362/) (2021)
- [A Risk-Sensitive Approach to Policy Optimization](https://mlsanity.com/paper/2208.09106/) (2022)
- [Policy Gradient Algorithms Implicitly Optimize by Continuation](https://mlsanity.com/paper/2305.06851/) (2023)
- [Guided Reinforcement Learning for Omnidirectional 3D Jumping in Quadruped Robots](https://mlsanity.com/paper/2507.16481/) (2025)
- [Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective](https://mlsanity.com/paper/2110.02421/) (2021)
- [Remember and Forget Experience Replay for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2203.13319/) (2022)
- [Clipped-Objective Policy Gradients for Pessimistic Policy Optimization](https://mlsanity.com/paper/2311.05846/) (2023)
- [Provably Efficient Convergence of Primal-Dual Actor-Critic with Nonlinear Function Approximation](https://mlsanity.com/paper/2202.13863/) (2022)
- [Distillation Policy Optimization](https://mlsanity.com/paper/2302.00533/) (2023)
- [Handling Cost and Constraints with Off-Policy Deep Reinforcement Learning](https://mlsanity.com/paper/2311.18684/) (2023)
- [Off-OAB: Off-Policy Policy Gradient Method with Optimal Action-Dependent Baseline](https://mlsanity.com/paper/2405.02572/) (2024)
- [Navigating Demand Uncertainty in Container Shipping: Deep Reinforcement Learning for Enabling Adaptive and Feasible Master Stowage Planning](https://mlsanity.com/paper/2502.12756/) (2025)
- [Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO](https://mlsanity.com/paper/2608.24488/) (2026)
- [End-to-End Safe Social Navigation via Multi-Task Reinforcement Learning and Probabilistic Perception](https://mlsanity.com/paper/2610.05733/) (2026)

## Abstract (arXiv)

Many continuous control tasks have bounded action spaces. When policy gradient methods are applied to such tasks, out-of-bound actions need to be clipped before execution, while policies are usually optimized as if the actions are not clipped. We propose a policy gradient estimator that exploits the knowledge of actions being clipped to reduce the variance in estimation. We prove that our estimator, named clipped action policy gradient (CAPG), is unbiased and achieves lower variance than the conventional estimator that ignores action bounds. Experimental results demonstrate that CAPG generally outperforms the conventional estimator, indicating that it is a better policy gradient estimator for continuous control tasks. The source code is available at https://github.com/pfnet-research/capg.