# Guided Policy Search as Approximate Mirror Descent

arXiv: https://arxiv.org/abs/1607.04614 · HTML: https://mlsanity.com/paper/1607.04614/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2016.

## Built on by (most cited first, 15 of 15)

- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization](https://mlsanity.com/paper/2501.17974/) (2025)
- [Neurosymbolic Transformers for Multi-Agent Communication](https://mlsanity.com/paper/2101.03238/) (2021)
- [Follow Your Path: a Progressive Method for Knowledge Distillation](https://mlsanity.com/paper/2107.09305/) (2021)
- [A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Search](https://mlsanity.com/paper/2202.07071/) (2022)
- [Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learning](https://mlsanity.com/paper/2209.07676/) (2022)
- [SPO: Sequential Monte Carlo Policy Optimisation](https://mlsanity.com/paper/2402.07963/) (2024)
- [DEALIO: Data-Efficient Adversarial Learning for Imitation from Observation](https://mlsanity.com/paper/2104.00163/) (2021)
- [Reinforcement Learning using Guided Observability](https://mlsanity.com/paper/2104.10986/) (2021)
- [Guided Policy Optimization under Partial Observability](https://mlsanity.com/paper/2505.15418/) (2025)
- [Homotopy Based Reinforcement Learning with Maximum Entropy for Autonomous Air Combat](https://mlsanity.com/paper/2112.01328/) (2021)
- [Recursive Least Squares Advantage Actor-Critic Algorithms](https://mlsanity.com/paper/2201.05918/) (2022)
- [Learning Solution Manifolds for Control Problems via Energy Minimization](https://mlsanity.com/paper/2203.03432/) (2022)
- [VAMPS: Visual and Motor Policies from Sampling-Based Planning](https://mlsanity.com/paper/2610.05331/) (2026)

## Abstract (arXiv)

Guided policy search algorithms can be used to optimize complex nonlinear policies, such as deep neural networks, without directly computing policy gradients in the high-dimensional parameter space. Instead, these methods use supervised learning to train the policy to mimic a "teacher" algorithm, such as a trajectory optimizer or a trajectory-centric reinforcement learning method. Guided policy search methods provide asymptotic local convergence guarantees by construction, but it is not clear how much the policy improves within a small, finite number of iterations. We show that guided policy search algorithms can be interpreted as an approximate variant of mirror descent, where the projection onto the constraint manifold is not exact. We derive a new guided policy search algorithm that is simpler and provides appealing improvement and convergence guarantees in simplified convex and linear settings, and show that in the more general nonlinear setting, the error in the projection step can be bounded. We provide empirical results on several simulated robotic navigation and manipulation tasks that show that our method is stable and achieves similar or better performance when compared to prior guided policy search methods, with a simpler formulation and fewer hyperparameters.