mlsanityData to 14 Oct 2026

Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithms

arXiv 2310.19927 ↗First posted 30 Oct 2023cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

ReParameterization (RP) Policy Gradient Methods (PGMs) have been widely adopted for continuous control tasks in robotics and computer graphics. However, recent studies have revealed that, when applied to long-term reinforcement learning problems, model-based RP PGMs may experience chaotic and non-smooth optimization landscapes with exploding gradient variance, which leads to slow convergence. This is in contrast to the conventional belief that reparameterization methods have low gradient estimation variance in problems such as training deep generative models. To comprehend this phenomenon, we conduct a theoretical examination of model-based RP PGMs and search for solutions to the optimization difficulties. Specifically, we analyze the convergence of the model-based RP PGMs and pinpoint the smoothness of function approximators as a major factor that affects the quality of gradient estimation. Based on our analysis, we propose a spectral normalization method to mitigate the exploding variance issue caused by long model unrolls. Our experimental results demonstrate that proper normalization significantly reduces the gradient variance of model-based RP PGMs. As a result, the performance of the proposed method is comparable or superior to other gradient estimators, such as the Likelihood Ratio (LR) gradient estimator. Our code is available at https://github.com/agentification/RP_PGM.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 45 of its 72 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

45 of 72 references.

References · 45

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  7. 2023Mastering Diverse Domains through World Modelscited by 1,253
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  10. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  11. 2019Model-Based Reinforcement Learning for Ataricited by 502
  12. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
33 more
  1. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  2. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  3. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  4. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  5. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  6. 2019Benchmarking Model-Based Reinforcement Learningcited by 162
  7. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  8. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  9. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  10. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  11. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  12. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 121
  13. 2016A Differentiable Physics Engine for Deep Learning in Roboticscited by 108
  14. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
  15. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  16. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 89
  17. 2021Gradients are Not All You Needcited by 86
  18. 2018Implicit Reparameterization Gradientscited by 85
  19. 2020NeuralSim: Augmenting Differentiable Simulators with Neural Networkscited by 82
  20. 2021DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cuttingcited by 64
  21. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 63
  22. 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 60
  23. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  24. 2021An End-to-End Differentiable Framework for Contact-Aware Robot Designcited by 54
  25. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  26. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 47
  27. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 42
  28. 2016The Generalized Reparameterization Gradientcited by 30
  29. 2021Bundled Gradients through Contact via Randomized Smoothingcited by 24
  30. 2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
  31. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  32. 2022Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learningcited by 5
  33. 2019Sample Complexity of Estimating the Policy Gradient for Nearly Deterministic Dynamical Systemscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2024How Can LLM Guide RL? A Value-Based Approachcited by 13
  2. 2024QuasiSim: Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfercited by 12
  3. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedbackcited by 11
  4. 2025Reparameterization Proximal Policy Optimizationcited by 1
  5. 2026Non-Myopic Active Feature Acquisition via Pathwise Policy Gradientscited by 1
  6. 2026Revisiting Mixture Policies in Entropy-Regularized Actor-Criticcited by 1
  7. 2025Distributions as Actions: A Unified Framework for Diverse Action Spacescited by 0