# Predictor-Corrector Policy Optimization

arXiv: https://arxiv.org/abs/1810.06509 · HTML: https://mlsanity.com/paper/1810.06509/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 15 Oct 2018.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Built on by (most cited first, 9 of 9)

- [Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning](https://mlsanity.com/paper/2502.02705/) (2025)
- [On-Policy Model Errors in Reinforcement Learning](https://mlsanity.com/paper/2110.07985/) (2021)
- [On Efficient Online Imitation Learning via Classification](https://mlsanity.com/paper/2209.12868/) (2022)
- [Functional Acceleration for Policy Mirror Descent](https://mlsanity.com/paper/2407.16602/) (2024)
- [Continuous Online Learning and New Insights to Online Imitation Learning](https://mlsanity.com/paper/1912.01261/) (2019)
- [Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning](https://mlsanity.com/paper/2412.07057/) (2024)
- [Explaining Fast Improvement in Online Imitation Learning](https://mlsanity.com/paper/2007.02520/) (2020)
- [Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms](https://mlsanity.com/paper/2312.16860/) (2023)
- [Acceleration in Policy Optimization](https://mlsanity.com/paper/2306.10587/) (2023)

## Abstract (arXiv)

We present a predictor-corrector framework, called PicCoLO, that can transform a first-order model-free reinforcement or imitation learning algorithm into a new hybrid method that leverages predictive models to accelerate policy learning. The new "PicCoLOed" algorithm optimizes a policy by recursively repeating two steps: In the Prediction Step, the learner uses a model to predict the unseen future gradient and then applies the predicted estimate to update the policy; in the Correction Step, the learner runs the updated policy in the environment, receives the true gradient, and then corrects the policy using the gradient error. Unlike previous algorithms, PicCoLO corrects for the mistakes of using imperfect predicted gradients and hence does not suffer from model bias. The development of PicCoLO is made possible by a novel reduction from predictable online learning to adversarial online learning, which provides a systematic way to modify existing first-order algorithms to achieve the optimal regret with respect to predictable information. We show, in both theory and simulation, that the convergence rate of several first-order model-free algorithms can be improved by PicCoLO.