mlsanityData to 14 Oct 2026

Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learning

arXiv 2311.15341 ↗First posted 26 Nov 2023cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Many problems in Reinforcement Learning (RL) seek an optimal policy with large discrete multidimensional yet unordered action spaces; these include problems in randomized allocation of resources such as placements of multiple security resources and emergency response units, etc. A challenge in this setting is that the underlying action space is categorical (discrete and unordered) and large, for which existing RL methods do not perform well. Moreover, these problems require validity of the realized action (allocation); this validity constraint is often difficult to express compactly in a closed mathematical form. The allocation nature of the problem also prefers stochastic optimal policies, if one exists. In this work, we address these challenges by (1) applying a (state) conditional normalizing flow to compactly represent the stochastic policy -- the compactness arises due to the network only producing one sampled action and the corresponding log probability of the action, which is then used by an actor-critic method; and (2) employing an invalid action rejection method (via a valid action oracle) to update the base policy. The action rejection is enabled by a modified policy gradient that we derive. Finally, we conduct extensive experiments to show the scalability of our approach compared to prior methods and the ability to enforce arbitrary state-conditional constraints on the support of the distribution of actions in any state.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 27 of its 35 references that are in the map. First posted on arXiv 26 Nov 2023.

Builds on

27 of 35 references.

References · 27

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2022Generative Adversarial Networkscited by 5,627
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2015Variational Inference with Normalizing Flowscited by 2,170
  6. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  7. 2015Importance Weighted Autoencoderscited by 456
  8. 2020PettingZoo: Gym for Multi-Agent Reinforcement Learningcited by 220
  9. 2021DC3: A learning method for optimization with hard constraintscited by 184
  10. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  11. 2020A Closer Look at Invalid Action Masking in Policy Gradient Algorithmscited by 171
  12. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 161
15 more
  1. 2017OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real Worldcited by 70
  2. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 68
  3. 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 46
  4. 2019Discrete and Continuous Action Representation for Practical RL in Video Gamescited by 39
  5. 2019Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policiescited by 38
  6. 2019Value constrained model-free continuous controlcited by 34
  7. 2018Resource Constrained Deep Reinforcement Learningcited by 24
  8. 2018Stochastic Gradient Descent with Biased but Consistent Gradient Estimatorscited by 18
  9. 2021Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimizationcited by 18
  10. 2019Solving Online Threat Screening Games using Constrained Action Space Reinforcement Learningcited by 10
  11. 2020A Review of Incident Prediction, Resource Allocation, and Dispatch Models for Emergency Managementcited by 9
  12. 2020Learning Stable Normalizing-Flow Control for Robotic Manipulationcited by 9
  13. 2021CLAIM: Curriculum Learning Policy for Influence Maximization in Unknown Social Networkscited by 7
  14. 2018Efficient Entropy for Policy Gradient with Multidimensional Action Spacecited by 5
  15. 2016The Mysteries of Security Games: Equilibrium Computation Becomes Combinatorial Algorithm Designcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 2.

10 papers.

  1. 2025Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPscited by 9
  2. 2025An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Taskscited by 5
  3. 2024Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learningcited by 4
  4. 2024BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spacescited by 3
  5. 2025Leveraging Constraint Violation Signals For Action-Constrained Reinforcement Learningcited by 2
  6. 2025Action-Constrained Imitation Learningcited by 2
  7. 2024Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policiescited by 1
  8. 2025SAINT: Attention-Based Policies for Discrete Combinatorial Action Spacescited by 0
  9. 2026Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initializationcited by 0
  10. 2026APC-RL: Exceeding Data-Driven Behavior Priors with Adaptive Policy Compositioncited by 0