# A survey on policy search algorithms for learning robot controllers in a handful of trials

arXiv: https://arxiv.org/abs/1807.02303 · HTML: https://mlsanity.com/paper/1807.02303/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2018.

## Built on by (most cited first, 30 of 35)

- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Robot Learning from Randomized Simulations: A Review](https://mlsanity.com/paper/2111.00956/) (2021)
- [Hierarchical Few-Shot Imitation with Skill Transition Models](https://mlsanity.com/paper/2107.08981/) (2021)
- [Policy Search for Model Predictive Control with Application to Agile Drone Flight](https://mlsanity.com/paper/2112.03850/) (2021)
- [Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation](https://mlsanity.com/paper/2107.01825/) (2021)
- [Recover: A Neuro-Symbolic Framework for Failure Detection and Recovery](https://mlsanity.com/paper/2404.00756/) (2024)
- [Reinforced In-Context Black-Box Optimization](https://mlsanity.com/paper/2402.17423/) (2024)
- [Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?](https://mlsanity.com/paper/2107.11587/) (2021)
- [Skill-based Multi-objective Reinforcement Learning of Industrial Robot Tasks with Planning and Knowledge Integration](https://mlsanity.com/paper/2203.10033/) (2022)
- [Hierarchical Planning for Complex Tasks with Knowledge Graph-RAG and Symbolic Verification](https://mlsanity.com/paper/2504.04578/) (2025)
- [A survey of Monte Carlo methods for noisy and costly densities with application to reinforcement learning and ABC](https://mlsanity.com/paper/2108.00490/) (2021)
- [Wasserstein Gradient Flows for Optimizing Gaussian Mixture Policies](https://mlsanity.com/paper/2305.10411/) (2023)
- [Online Damage Recovery for Physical Robots with Hierarchical Quality-Diversity](https://mlsanity.com/paper/2210.09918/) (2022)
- [Performance-driven Constrained Optimal Auto-Tuner for MPC](https://mlsanity.com/paper/2503.07127/) (2025)
- [Hierarchical Quality-Diversity for Online Damage Recovery](https://mlsanity.com/paper/2204.05726/) (2022)
- [Learning of Parameters in Behavior Trees for Movement Skills](https://mlsanity.com/paper/2109.13050/) (2021)
- [ReVoLT: Relational Reasoning and Voronoi Local Graph Planning for Target-driven Navigation](https://mlsanity.com/paper/2301.02382/) (2023)
- [The Mini Wheelbot: A Testbed for Learning-based Balancing, Flips, and Articulated Driving](https://mlsanity.com/paper/2502.04582/) (2025)
- [On the Statistical Query Complexity of Learning Semiautomata: a Random Walk Approach](https://mlsanity.com/paper/2510.04115/) (2025)
- [Local Bayesian Optimization for Controller Tuning with Crash Constraints](https://mlsanity.com/paper/2411.16267/) (2024)
- [Optimality in Noisy Importance Sampling](https://mlsanity.com/paper/2201.02432/) (2022)
- [Modular Adaptive Policy Selection for Multi-Task Imitation Learning through Task Division](https://mlsanity.com/paper/2203.14855/) (2022)
- [End-to-End Stable Imitation Learning via Autonomous Neural Dynamic Policies](https://mlsanity.com/paper/2305.12886/) (2023)
- [Learning to Exploit Elastic Actuators for Quadruped Locomotion](https://mlsanity.com/paper/2209.07171/) (2022)
- [E2R: a Hierarchical-Learning inspired Novelty-Search method to generate diverse repertoires of grasping trajectories](https://mlsanity.com/paper/2210.07887/) (2022)
- [Dropout Strategy in Reinforcement Learning: Limiting the Surrogate Objective Variance in Policy Optimization Methods](https://mlsanity.com/paper/2310.20380/) (2023)
- [Adaptable Recovery Behaviors in Robotics: A Behavior Trees and Motion Generators(BTMG) Approach for Failure Management](https://mlsanity.com/paper/2404.06129/) (2024)
- [On the Fly Adaptation of Behavior Tree-Based Policies through Reinforcement Learning](https://mlsanity.com/paper/2503.06309/) (2025)
- [Social Learning Strategies for Evolved Virtual Soft Robots](https://mlsanity.com/paper/2604.12482/) (2026)
- [GPU-Accelerated Policy Optimization via Batch Automatic Differentiation of Gaussian Processes for Real-World Control](https://mlsanity.com/paper/2202.13638/) (2022)

## Abstract (arXiv)

Most policy search algorithms require thousands of training episodes to find an effective policy, which is often infeasible with a physical robot. This survey article focuses on the extreme other end of the spectrum: how can a robot adapt with only a handful of trials (a dozen) and a few minutes? By analogy with the word "big-data", we refer to this challenge as "micro-data reinforcement learning". We show that a first strategy is to leverage prior knowledge on the policy structure (e.g., dynamic movement primitives), on the policy parameters (e.g., demonstrations), or on the dynamics (e.g., simulators). A second strategy is to create data-driven surrogate models of the expected reward (e.g., Bayesian optimization) or the dynamical model (e.g., model-based policy search), so that the policy optimizer queries the model instead of the real system. Overall, all successful micro-data algorithms combine these two strategies by varying the kind of model and prior knowledge. The current scientific challenges essentially revolve around scaling up to complex robots (e.g., humanoids), designing generic priors, and optimizing the computing time.