# Path Integral Guided Policy Search

arXiv: https://arxiv.org/abs/1610.00529 · HTML: https://mlsanity.com/paper/1610.00529/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2016.

## Built on by (most cited first, 15 of 15)

- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Reinforcement and Imitation Learning for Diverse Visuomotor Skills](https://mlsanity.com/paper/1802.09564/) (2018)
- [Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention](https://mlsanity.com/paper/2104.11203/) (2021)
- [COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning](https://mlsanity.com/paper/2010.14500/) (2020)
- [Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation](https://mlsanity.com/paper/2107.13545/) (2021)
- [Learning Modular Robot Control Policies](https://mlsanity.com/paper/2105.10049/) (2021)
- [Reinforcement Learning with Latent Flow](https://mlsanity.com/paper/2101.01857/) (2021)
- [Active Learning of Discrete-Time Dynamics for Uncertainty-Aware Model Predictive Control](https://mlsanity.com/paper/2210.12583/) (2022)
- [ModelLight: Model-Based Meta-Reinforcement Learning for Traffic Signal Control](https://mlsanity.com/paper/2111.08067/) (2021)
- [Bootstrapping Motor Skill Learning with Motion Planning](https://mlsanity.com/paper/2101.04736/) (2021)
- [Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradient](https://mlsanity.com/paper/2605.26478/) (2026)
- [Analyzing the Hidden Activations of Deep Policy Networks: Why Representation Matters](https://mlsanity.com/paper/2103.06398/) (2021)
- [Temporal Shift Reinforcement Learning](https://mlsanity.com/paper/2109.02145/) (2021)
- [EVaDE : Event-Based Variational Thompson Sampling for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2501.09611/) (2025)
- [VAMPS: Visual and Motor Policies from Sampling-Based Planning](https://mlsanity.com/paper/2610.05331/) (2026)

## Abstract (arXiv)

We present a policy search method for learning complex feedback control policies that map from high-dimensional sensory inputs to motor torques, for manipulation tasks with discontinuous contact dynamics. We build on a prior technique called guided policy search (GPS), which iteratively optimizes a set of local policies for specific instances of a task, and uses these to train a complex, high-dimensional global policy that generalizes across task instances. We extend GPS in the following ways: (1) we propose the use of a model-free local optimizer based on path integral stochastic optimal control (PI2), which enables us to learn local policies for tasks with highly discontinuous contact dynamics; and (2) we enable GPS to train on a new set of task instances in every iteration by using on-policy sampling: this increases the diversity of the instances that the policy is trained on, and is crucial for achieving good generalization. We show that these contributions enable us to learn deep neural network policies that can directly perform torque control from visual input. We validate the method on a challenging door opening task and a pick-and-place task, and we demonstrate that our approach substantially outperforms the prior LQR-based local policy optimizer on these tasks. Furthermore, we show that on-policy sampling significantly increases the generalization ability of these policies.