# Improving Stochastic Action-Constrained Reinforcement Learning via Truncated Distributions

arXiv: https://arxiv.org/abs/2511.22406 · HTML: https://mlsanity.com/paper/2511.22406/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 27 Nov 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking](https://mlsanity.com/paper/2205.06750/) (2022)
- [Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking](https://mlsanity.com/paper/2406.03704/) (2024)
- [Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization](https://mlsanity.com/paper/2102.11055/) (2021)
- [FlowPG: Action-constrained Policy Gradient with Normalizing Flows](https://mlsanity.com/paper/2402.05149/) (2024)
- [Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraints](https://mlsanity.com/paper/2304.08743/) (2023)
- [Solving Online Threat Screening Games using Constrained Action Space Reinforcement Learning](https://mlsanity.com/paper/1911.08799/) (2019)
- [Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs](https://mlsanity.com/paper/2503.12932/) (2025)
- [Volesti: Volume Approximation and Sampling for Convex Polytopes in R](https://mlsanity.com/paper/2007.01578/) (2020)
- [Learning to Generate All Feasible Actions](https://mlsanity.com/paper/2301.11461/) (2023)
- [Leveraging Constraint Violation Signals For Action-Constrained Reinforcement Learning](https://mlsanity.com/paper/2502.10431/) (2025)

## Built on by (most cited first, 2 of 2)

- [Constraint-Enhanced Reinforcement Learning Based on Dynamic Decoupled Spherical Radial Squashing](https://mlsanity.com/paper/2605.04185/) (2026)
- [The Geometry of Randomized Smoothing on Feasible Sets](https://mlsanity.com/paper/2609.39497/) (2026)

## Abstract (arXiv)

In reinforcement learning (RL), it is often advantageous to consider additional constraints on the action space to ensure safety or action relevance. Existing work on such action-constrained RL faces challenges regarding effective policy updates, computational efficiency, and predictable runtime. Recent work proposes to use truncated normal distributions for stochastic policy gradient methods. However, the computation of key characteristics, such as the entropy, log-probability, and their gradients, becomes intractable under complex constraints. Hence, prior work approximates these using the non-truncated distributions, which severely degrades performance. We argue that accurate estimation of these characteristics is crucial in the action-constrained RL setting, and propose efficient numerical approximations for them. We also provide an efficient sampling strategy for truncated policy distributions and validate our approach on three benchmark environments, which demonstrate significant performance improvements when using accurate estimations.