# Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning

arXiv: https://arxiv.org/abs/2005.10872 · HTML: https://mlsanity.com/paper/2005.10872/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2020.

## Built on by (most cited first, 25 of 25)

- [What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2204.06252/) (2022)
- [Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks](https://mlsanity.com/paper/2110.03655/) (2021)
- [Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction](https://mlsanity.com/paper/2409.18121/) (2024)
- [Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks](https://mlsanity.com/paper/2405.01534/) (2024)
- [Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Image](https://mlsanity.com/paper/2109.06161/) (2021)
- [Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learning](https://mlsanity.com/paper/2204.02863/) (2022)
- [Affordance Learning from Play for Sample-Efficient Policy Learning](https://mlsanity.com/paper/2203.00352/) (2022)
- [Residual Learning from Demonstration: Adapting DMPs for Contact-rich Manipulation](https://mlsanity.com/paper/2008.07682/) (2020)
- [Goal-conditioned dual-action imitation learning for dexterous dual-arm robot manipulation](https://mlsanity.com/paper/2203.09749/) (2022)
- [Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics](https://mlsanity.com/paper/2107.09822/) (2021)
- [Back to the Manifold: Recovering from Out-of-Distribution States](https://mlsanity.com/paper/2207.08673/) (2022)
- [InsertionNet -- A Scalable Solution for Insertion](https://mlsanity.com/paper/2104.14223/) (2021)
- [On the Effectiveness of Retrieval, Alignment, and Replay in Manipulation](https://mlsanity.com/paper/2312.12345/) (2023)
- [Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies](https://mlsanity.com/paper/2107.08942/) (2021)
- [Emergent Dexterity via Diverse Resets and Large-Scale Reinforcement Learning](https://mlsanity.com/paper/2603.15789/) (2026)
- [Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spaces](https://mlsanity.com/paper/2105.11283/) (2021)
- [InsertionNet 2.0: Minimal Contact Multi-Step Insertion Using Multimodal Multiview Sensory Input](https://mlsanity.com/paper/2203.01153/) (2022)
- [PLANRL: A Motion Planning and Imitation Learning Framework to Bootstrap Reinforcement Learning](https://mlsanity.com/paper/2408.04054/) (2024)
- [E2HiL: Entropy-Guided Sample Selection for Efficient Real-World Human-in-the-Loop Reinforcement Learning](https://mlsanity.com/paper/2601.19969/) (2026)
- [Combining Learning from Demonstration with Learning by Exploration to Facilitate Contact-Rich Tasks](https://mlsanity.com/paper/2103.05904/) (2021)
- [Efficient Skill Acquisition for Complex Manipulation Tasks in Obstructed Environments](https://mlsanity.com/paper/2303.03365/) (2023)
- [FLEX: A Framework for Learning Robot-Agnostic Force-based Skills Involving Sustained Contact Object Manipulation](https://mlsanity.com/paper/2503.13418/) (2025)
- [Failure Forecasting Boosts Robustness of Sim2Real Rhythmic Insertion Policies](https://mlsanity.com/paper/2507.06519/) (2025)
- [Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/2606.03949/) (2026)
- [Cognitive Manipulation: Semi-supervised Visual Representation and Classroom-to-real Reinforcement Learning for Assembly in Semi-structured Environments](https://mlsanity.com/paper/2406.00364/) (2024)

## Abstract (arXiv)

Traditional robotic approaches rely on an accurate model of the environment, a detailed description of how to perform the task, and a robust perception system to keep track of the current state. On the other hand, reinforcement learning approaches can operate directly from raw sensory inputs with only a reward signal to describe the task, but are extremely sample-inefficient and brittle. In this work, we combine the strengths of model-based methods with the flexibility of learning-based methods to obtain a general method that is able to overcome inaccuracies in the robotics perception/actuation pipeline, while requiring minimal interactions with the environment. This is achieved by leveraging uncertainty estimates to divide the space in regions where the given model-based policy is reliable, and regions where it may have flaws or not be well defined. In these uncertain regions, we show that a locally learned-policy can be used directly with raw sensory inputs. We test our algorithm, Guided Uncertainty-Aware Policy Optimization (GUAPO), on a real-world robot performing peg insertion. Videos are available at https://sites.google.com/view/guapo-rl