mlsanityData to 7 Oct 2026

PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos

arXiv 1902.01240 ↗First posted 4 Feb 2019cs.LG cs.AI cs.ROcited by 56

Not on the map. Why?

Abstract (arXiv)

Previously, the exploding gradient problem has been explained to be central in deep learning and model-based reinforcement learning, because it causes numerical issues and instability in optimization. Our experiments in model-based reinforcement learning imply that the problem is not just a numerical issue, but it may be caused by a fundamental chaos-like nature of long chains of nonlinear computations. Not only do the magnitudes of the gradients become large, the direction of the gradients becomes essentially random. We show that reparameterization gradients suffer from the problem, while likelihood ratio gradients are robust. Using our insights, we develop a model-based policy search framework, Probabilistic Inference for Particle-Based Policy Search (PIPPS), which is easily extensible, and allows for almost arbitrary models and policies, while simultaneously matching the performance of previous data-efficient learning algorithms. Finally, we invent the total propagation algorithm, which efficiently computes a union over all pathwise derivative depths during a single backwards pass, automatically giving greater weight to estimators with lower variance, sometimes improving over reparameterization gradients by 106 times.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 10; 2022: 6; 2023: 13; 2024: 8; 2025: 8; 2026: 9.

56 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  3. 2023Denoising Diffusion Samplerscited by 144
  4. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 92
  5. 2022Tutorial on amortized optimizationcited by 83
  6. 2021Gradients are Not All You Needcited by 82
  7. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  8. 2025What Makes a Good Diffusion Planner for Decision Making?cited by 37
  9. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  10. 2024Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulationcited by 18
  11. 2021Model-based Meta Reinforcement Learning using Graph Structured Surrogate Modelscited by 16
  12. 2022Symbolic Learning to Optimize: Towards Interpretability and Scalabilitycited by 16
38 more
  1. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  2. 2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 14
  3. 2021A unified view of likelihood ratio and reparameterization gradientscited by 12
  4. 2023Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Controlcited by 11
  5. 2021PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulatorscited by 10
  6. 2024Do Transformer World Models Give Better Policy Gradients?cited by 10
  7. 2021Model-free Policy Learning with Reward Gradientscited by 9
  8. 2021Mixed Policy Gradient: off-policy reinforcement learning driven jointly by data and modelcited by 8
  9. 2023Fractal Landscapes in Policy Optimizationcited by 8
  10. 2023Gradient Informed Proximal Policy Optimizationcited by 8
  11. 2021Model-Based Policy Search Using Monte Carlo Gradient Estimation with Real Systems Applicationcited by 7
  12. 2023Low-Variance Gradient Estimation in Unrolled Computation Graphs with ES-Singlecited by 7
  13. 2023Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithmscited by 7
  14. 2024μLO: Compute-Efficient Meta-Generalization of Learned Optimizerscited by 7
  15. 2023Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learningcited by 5
  16. 2023Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategiescited by 5
  17. 2024Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approachcited by 4
  18. 2025Time-Aware World Model for Adaptive Prediction and Controlcited by 4
  19. 2023Toward Open-ended Embodied Tasks Solvingcited by 3
  20. 2024Stabilizing Backpropagation Through Time to Learn Complex Physicscited by 3
  21. 2023DiSProD: Differentiable Symbolic Propagation of Distributions for Planningcited by 2
  22. 2024Mollification Effects of Policy Gradient Methodscited by 2
  23. 2025Improving Controller Generalization with Dimensionless Markov Decision Processescited by 2
  24. 2026On Advantage Estimates for Max@K Policy Gradientscited by 2
  25. 2021Learning Robust Controllers Via Probabilistic Model-Based Policy Searchcited by 1
  26. 2022Reinforcement Learning for Personalized Drug Discovery and Design for Complex Diseases: A Systems Pharmacology Perspectivecited by 1
  27. 2025Accelerated Learning with Linear Temporal Logic using Differentiable Simulationcited by 1
  28. 2025Reparameterization Proximal Policy Optimizationcited by 1
  29. 2025Double Horizon Model-Based Policy Optimizationcited by 1
  30. 2026Dream-MPC: Gradient-Based Model Predictive Control with Latent Imaginationcited by 1
  31. 2026Retry Policy Gradients in Continuous Action Spacescited by 1
  32. 2026OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimationcited by 1
  33. 2021Model-based Policy Search for Partially Measurable Systemscited by 0
  34. 2022Integrated Decision and Control for High-Level Automated Vehicles by Mixed Policy Gradient and Its Experiment Verificationcited by 0
  35. 2023Enabling Efficient, Reliable Real-World Reinforcement Learning with Approximate Physics-Based Modelscited by 0
  36. 2023Physics-informed reinforcement learning via probabilistic co-adjustment functionscited by 0
  37. 2024Diminishing Return of Value Expansion Methodscited by 0
  38. 2025Accelerating Model-Based Reinforcement Learning using Non-Linear Trajectory Optimizationcited by 0

50 most cited of 56 All 56 (JSON)