mlsanityData to 7 Oct 2026

Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation

arXiv 2605.18591 ↗First posted 18 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Natural policy gradients improve optimization by accounting for the geometry of distribution space, but their practical use is limited by the cost of estimating and inverting the Fisher matrix. We present Randomized Advantage Transformation (RAT), a method for estimating Tikhonov-regularized natural policy gradients via direct backpropagation. By applying the Woodbury formula, we reformulate the regularized natural policy gradients as vanilla policy gradients with a transformed advantage. RAT computes this transformation efficiently via randomized block Kaczmarz iterations on on-policy mini-batches, avoiding explicit Fisher construction, conjugate-gradient solvers, and architecture-specific approximations. We provide convergence guarantees for RAT and demonstrate empirically that it matches or exceeds established natural-gradient methods across continuous and visual control benchmarks, while remaining simple to implement and compatible with various architectures.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

29 of 46 references.

References · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2016OpenAI Gymcited by 1,684
  8. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  9. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  10. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  11. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  12. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
17 more
  1. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  2. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  3. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  4. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 143
  5. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
  6. 2013Revisiting the Nystrom Method for Improved Large-Scale Machine Learningcited by 76
  7. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  8. 2021Batch size-invariance for policy optimizationcited by 33
  9. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29
  10. 2024A Kaczmarz-inspired approach to accelerate the optimization of neural network wavefunctionscited by 24
  11. 2021ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurecited by 16
  12. 2024An Improved Empirical Fisher Approximation for Natural Gradient Descentcited by 15
  13. 2025Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomizationcited by 13
  14. 2022Trust Region Bounds for Decentralized PPO Under Non-stationaritycited by 11
  15. 2024Recurrent Natural Policy Gradient for POMDPscited by 8
  16. 2025A Sketch-and-Project Analysis of Subsampled Natural Gradient Algorithmscited by 2
  17. 2026Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learningcited by 2

Built on by

Not cited yet.