# Model-Based Policy Search Using Monte Carlo Gradient Estimation with Real Systems Application

arXiv: https://arxiv.org/abs/2101.12115 · HTML: https://mlsanity.com/paper/2101.12115/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 28 Jan 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Gaussian Processes for Data-Efficient Learning in Robotics and Control](https://mlsanity.com/paper/1502.02860/) (2015)
- [PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos](https://mlsanity.com/paper/1902.01240/) (2019)
- [Shaping the learning landscape in neural networks around wide flat minima](https://mlsanity.com/paper/1905.07833/) (2019)
- [PEGASUS: A Policy Search Method for Large MDPs and POMDPs](https://mlsanity.com/paper/1301.3878/) (2013)
- [Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations](https://mlsanity.com/paper/1907.07578/) (2019)
- [Black-Box Data-efficient Policy Search for Robotics](https://mlsanity.com/paper/1703.07261/) (2017)
- [Derivative-free online learning of inverse dynamics models](https://mlsanity.com/paper/1809.05074/) (2018)
- [Semiparametrical Gaussian Processes Learning of Forward Dynamical Models for Navigating in a Circular Maze](https://mlsanity.com/paper/1809.04993/) (2018)
- [Online semi-parametric learning for inverse dynamics modeling](https://mlsanity.com/paper/1603.05412/) (2016)
- [Model-Based Reinforcement Learning for Physical Systems Without Velocity and Acceleration Measurements](https://mlsanity.com/paper/2002.10621/) (2020)
- [A novel Multiplicative Polynomial Kernel for Volterra series identification](https://mlsanity.com/paper/1905.07960/) (2019)

## Built on by (most cited first, 10 of 10)

- [Learning Control from Raw Position Measurements](https://mlsanity.com/paper/2301.13183/) (2023)
- [DiSProD: Differentiable Symbolic Propagation of Distributions for Planning](https://mlsanity.com/paper/2302.01491/) (2023)
- [AI Olympics challenge with Evolutionary Soft Actor Critic](https://mlsanity.com/paper/2409.01104/) (2024)
- [Learning control of underactuated double pendulum with Model-Based Reinforcement Learning](https://mlsanity.com/paper/2409.05811/) (2024)
- [Reinforcement Learning for Robust Athletic Intelligence: Lessons from the 2nd 'AI Olympics with RealAIGym' Competition](https://mlsanity.com/paper/2503.15290/) (2025)
- [Controlled Gaussian Process Dynamical Models with Application to Robotic Cloth Manipulation](https://mlsanity.com/paper/2103.06615/) (2021)
- [FORESEE: Prediction with Expansion-Compression Unscented Transform for Online Policy Optimization](https://mlsanity.com/paper/2209.12644/) (2022)
- [Learning global control of underactuated systems with Model-Based Reinforcement Learning](https://mlsanity.com/paper/2504.06721/) (2025)
- [Accelerating Model-Based Reinforcement Learning using Non-Linear Trajectory Optimization](https://mlsanity.com/paper/2506.02767/) (2025)
- [Finetuning Deep Reinforcement Learning Policies with Evolutionary Strategies for Control of Underactuated Robots](https://mlsanity.com/paper/2507.10030/) (2025)

## Abstract (arXiv)

In this paper, we present a Model-Based Reinforcement Learning (MBRL) algorithm named Monte Carlo Probabilistic Inference for Learning COntrol (MC-PILCO). The algorithm relies on Gaussian Processes (GPs) to model the system dynamics and on a Monte Carlo approach to estimate the policy gradient. This defines a framework in which we ablate the choice of the following components: (i) the selection of the cost function, (ii) the optimization of policies using dropout, (iii) an improved data efficiency through the use of structured kernels in the GP models. The combination of the aforementioned aspects affects dramatically the performance of MC-PILCO. Numerical comparisons in a simulated cart-pole environment show that MC-PILCO exhibits better data efficiency and control performance w.r.t. state-of-the-art GP-based MBRL algorithms. Finally, we apply MC-PILCO to real systems, considering in particular systems with partially measurable states. We discuss the importance of modeling both the measurement system and the state estimators during policy optimization. The effectiveness of the proposed solutions has been tested in simulation and on two real systems, a Furuta pendulum and a ball-and-plate rig.