mlsanityData to 14 Oct 2026

Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization

arXiv 2102.11055 ↗First posted 22 Feb 2021cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Action-constrained reinforcement learning (RL) is a widely-used approach in various real-world applications, such as scheduling in networked systems with resource constraints and control of a robot with kinematic constraints. While the existing projection-based approaches ensure zero constraint violation, they could suffer from the zero-gradient problem due to the tight coupling of the policy gradient and the projection, which results in sample-inefficient training and slow convergence. To tackle this issue, we propose a learning algorithm that decouples the action constraints from the policy parameter update by leveraging state-wise Frank-Wolfe and a regression-based policy update scheme. Moreover, we show that the proposed algorithm enjoys convergence and policy improvement properties in the tabular case as well as generalizes the popular DDPG algorithm for action-constrained RL in the general case. Through experiments, we demonstrate that the proposed algorithm significantly outperforms the benchmark methods on a variety of control tasks.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

31 of 46 references.

References · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  7. 2017Constrained Policy Optimizationcited by 983
  8. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  9. 2014Convex Optimization: Algorithms and Complexitycited by 575
  10. 2019Differentiable Convex Optimization Layerscited by 465
  11. 2018Reward Constrained Policy Optimizationcited by 417
  12. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
19 more
  1. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  2. 2018Safe Exploration in Continuous Action Spacescited by 247
  3. 2020Projection-Based Constrained Policy Optimizationcited by 203
  4. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  5. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  6. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  7. 2017OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real Worldcited by 70
  8. 2016Convergence Rate of Frank-Wolfe for Non-Convex Objectivescited by 63
  9. 2019ChainerRL: A Deep Reinforcement Learning Librarycited by 51
  10. 2019DeepGait: Planning and Control of Quadrupedal Gaits using Deep Reinforcement Learningcited by 50
  11. 2016Stochastic Frank-Wolfe Methods for Nonconvex Optimizationcited by 41
  12. 2018Experience-driven Networking: A Deep Reinforcement Learning based Approachcited by 41
  13. 2020Constrained Markov Decision Processes via Backward Value Functionscited by 33
  14. 2018Resource Constrained Deep Reinforcement Learningcited by 24
  15. 2019Robust Regression for Safe Exploration in Controlcited by 24
  16. 2018Clipped Action Policy Gradientcited by 19
  17. 2020CFR-RL: Traffic Engineering with Reinforcement Learning in SDNcited by 12
  18. 2019Solving Online Threat Screening Games using Constrained Action Space Reinforcement Learningcited by 10
  19. 2018Internet Congestion Control via Deep Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 4; 2024: 2; 2025: 6; 2026: 3.

18 papers.

Sort:

By area:

  1. 2021Breaking the Linear Iteration Cost Barrier for Some Well-known Conditional Gradient Methods Using MaxIP Data-structurescited by 19
  2. 2024FlowPG: Action-constrained Policy Gradient with Normalizing Flowscited by 16
  3. 2023Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraintscited by 11
  4. 2023Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learningcited by 10
  5. 2025Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPscited by 9
  6. 2025Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?cited by 5
  7. 2023DFWLayer: Differentiable Frank-Wolfe Optimization Layercited by 2
  8. 2025Leveraging Constraint Violation Signals For Action-Constrained Reinforcement Learningcited by 2
  9. 2025Action-Constrained Imitation Learningcited by 2
  10. 2025Improving Stochastic Action-Constrained Reinforcement Learning via Truncated Distributionscited by 2
  11. 2022Neural Frank-Wolfe Policy Optimization for Region-of-Interest Intra-Frame Coding with HEVC/H.265cited by 1
  12. 2026Semi-Supervised Cross-Domain Imitation Learningcited by 1
6 more
  1. 2026Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learningcited by 1
  2. 2026Differentiable Policy Transport over Multi-Layer Network Feasibility Geometrycited by 1
  3. 2022Action-Constrained Reinforcement Learning for Frame-Level Bit Allocation in HEVC/H.265 through Frank-Wolfe Policy Optimizationcited by 0
  4. 2023Nonsmooth Projection-Free Optimization with Functional Constraintscited by 0
  5. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraintscited by 0
  6. 2025Navigating Demand Uncertainty in Container Shipping: Deep Reinforcement Learning for Enabling Adaptive and Feasible Master Stowage Planningcited by 0