mlsanityData to 14 Oct 2026

KFCPO: Kronecker-Factored Approximated Constrained Policy Optimization

arXiv 2511.00880 ↗First posted 2 Nov 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

We propose KFCPO, a novel Safe Reinforcement Learning (Safe RL) algorithm that combines scalable Kronecker-Factored Approximate Curvature (K-FAC) based second-order policy optimization with safety-aware gradient manipulation. KFCPO leverages K-FAC to perform efficient and stable natural gradient updates by approximating the Fisher Information Matrix (FIM) in a layerwise, closed form manner, avoiding iterative approximation overheads. To address the tradeoff between reward maximization and constraint satisfaction, we introduce a margin aware gradient manipulation mechanism that adaptively adjusts the influence of reward and cost gradients based on the agent's proximity to safety boundaries. This method blends gradients using a direction sensitive projection, eliminating harmful interference and avoiding abrupt changes caused by fixed hard thresholds. Additionally, a minibatch level KL rollback strategy is adopted to ensure trust region compliance and to prevent destabilizing policy shifts. Experiments on Safety Gymnasium using OmniSafe show that KFCPO achieves 10.3% to 50.2% higher average return across environments compared to the best baseline that respected the safety constraint, demonstrating superior balance of safety and performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 14 of its 22 references that are in the map. First posted on arXiv 2 Nov 2025.

Builds on

14 of 22 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation2 May

Other references · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  4. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  5. 2017Constrained Policy Optimizationcited by 983
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  7. 2020Projection-Based Constrained Policy Optimizationcited by 203
  8. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  9. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  10. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 121
  11. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  12. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
1 more
  1. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 58

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
  2. 2025SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints
  3. 2025Anytime Safe Reinforcement Learning
  4. 2026SafePG: Safe and Globally Optimal Reinforcement Learning with Hard Constraints
  5. 2025Proactive Constrained Policy Optimization with Preemptive Penalty
  6. 2025Off-Policy Reinforcement Learning with Anytime Safety Guarantees via Robust Safe Gradient Flow
  7. 2024Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learning
  8. 2026Boundary-Seeking Policy Gradient for Safe Reinforcement Learning