# Practice Makes Perfect: Planning to Learn Skill Parameter Policies

arXiv: https://arxiv.org/abs/2402.15025 · HTML: https://mlsanity.com/paper/2402.15025/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 67 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Builds on (most cited first, 30 of 42)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [Self-Supervised Exploration via Disagreement](https://mlsanity.com/paper/1906.04161/) (2019)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Parrot: Data-Driven Behavioral Priors for Reinforcement Learning](https://mlsanity.com/paper/2011.10024/) (2020)
- [Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention](https://mlsanity.com/paper/2104.11203/) (2021)
- [Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models](https://mlsanity.com/paper/2401.03360/) (2023)
- [CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/1810.06284/) (2018)
- [Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks](https://mlsanity.com/paper/2110.03655/) (2021)
- [A Survey of Exploration Methods in Reinforcement Learning](https://mlsanity.com/paper/2109.00157/) (2021)
- [Deep Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/1511.04143/) (2015)
- [LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discovery](https://mlsanity.com/paper/2311.02058/) (2023)
- [Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives](https://mlsanity.com/paper/2110.15360/) (2021)
- [Reinforcement Learning with Parameterized Actions](https://mlsanity.com/paper/1509.01644/) (2015)
- [Online Replanning in Belief Space for Partially Observable Task and Motion Problems](https://mlsanity.com/paper/1911.04577/) (2019)
- [Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots](https://mlsanity.com/paper/1301.4862/) (2013)
- [Learning Symbolic Operators for Task and Motion Planning](https://mlsanity.com/paper/2103.00589/) (2021)
- [Learning Neuro-Symbolic Skills for Bilevel Planning](https://mlsanity.com/paper/2206.10680/) (2022)
- [Predicate Invention for Bilevel Planning](https://mlsanity.com/paper/2203.09634/) (2022)
- [Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image](https://mlsanity.com/paper/2006.05398/) (2020)
- [Learning compositional models of robot skills for task and motion planning](https://mlsanity.com/paper/2006.06444/) (2020)
- [Compositional Diffusion-Based Continuous Constraint Solvers](https://mlsanity.com/paper/2309.00966/) (2023)
- [Simple regret for infinitely many armed bandits](https://mlsanity.com/paper/1505.04627/) (2015)
- [Reset-Free Lifelong Learning with Skill-Space Planning](https://mlsanity.com/paper/2012.03548/) (2020)

## Built on by (most cited first, 23 of 23)

- [A Self-Improving Coding Agent](https://mlsanity.com/paper/2504.15228/) (2025)
- [A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning](https://mlsanity.com/paper/2509.15937/) (2025)
- [MaestroMotif: Skill Design from Artificial Intelligence Feedback](https://mlsanity.com/paper/2412.08542/) (2024)
- [Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfaction](https://mlsanity.com/paper/2406.05572/) (2024)
- [From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Models](https://mlsanity.com/paper/2501.00296/) (2024)
- [Adaptive Language-Guided Abstraction from Contrastive Explanations](https://mlsanity.com/paper/2409.08212/) (2024)
- [On-Robot Reinforcement Learning with Goal-Contrastive Rewards](https://mlsanity.com/paper/2410.19989/) (2024)
- [MAGELLAN: Metacognitive predictions of learning progress guide autotelic LLM agents in large goal spaces](https://mlsanity.com/paper/2502.07709/) (2025)
- [Equivariant Action Sampling for Reinforcement Learning and Planning](https://mlsanity.com/paper/2412.12237/) (2024)
- [A Systematic Study of Large Language Models for Task and Motion Planning With PDDLStream](https://mlsanity.com/paper/2510.00182/) (2025)
- [SLAP: Shortcut Learning for Abstract Planning](https://mlsanity.com/paper/2511.01107/) (2025)
- [Learning Long-Horizon Robot Manipulation Skills via Privileged Action](https://mlsanity.com/paper/2502.15442/) (2025)
- [Shadow Program Inversion with Differentiable Planning: A Framework for Unified Robot Program Parameter and Trajectory Optimization](https://mlsanity.com/paper/2409.08678/) (2024)
- [Learning to Bridge the Gap: Efficient Novelty Recovery with Planning and Reinforcement Learning](https://mlsanity.com/paper/2409.19226/) (2024)
- [Automating Curriculum Learning for Reinforcement Learning using a Skill-Based Bayesian Network](https://mlsanity.com/paper/2502.15662/) (2025)
- [MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systems](https://mlsanity.com/paper/2504.17070/) (2025)
- [Coloring Between the Lines: Personalization in the Null Space of Planning Constraints](https://mlsanity.com/paper/2505.15503/) (2025)
- [Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models](https://mlsanity.com/paper/2609.32069/) (2026)
- [ActivePusher: Active Learning and Planning with Residual Physics for Nonprehensile Manipulation](https://mlsanity.com/paper/2506.04646/) (2025)
- [Empowering LLMs with Parameterized Skills for Adversarial Long-Horizon Planning](https://mlsanity.com/paper/2509.13127/) (2025)
- [Neuro-Symbolic Learning for Long-Horizon Task Planning Under Complex Logical Constraints](https://mlsanity.com/paper/2606.06877/) (2026)
- [HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentation](https://mlsanity.com/paper/2607.09776/) (2026)
- [Deliberate Practice: Learning Robot Skills under a Budget](https://mlsanity.com/paper/2608.13415/) (2026)

## Abstract (arXiv)

One promising approach towards effective robot decision making in complex, long-horizon tasks is to sequence together parameterized skills. We consider a setting where a robot is initially equipped with (1) a library of parameterized skills, (2) an AI planner for sequencing together the skills given a goal, and (3) a very general prior distribution for selecting skill parameters. Once deployed, the robot should rapidly and autonomously learn to improve its performance by specializing its skill parameter selection policy to the particular objects, goals, and constraints in its environment. In this work, we focus on the active learning problem of choosing which skills to practice to maximize expected future task success. We propose that the robot should estimate the competence of each skill, extrapolate the competence (asking: "how much would the competence improve through practice?"), and situate the skill in the task distribution through competence-aware planning. This approach is implemented within a fully autonomous system where the robot repeatedly plans, practices, and learns without any environment resets. Through experiments in simulation, we find that our approach learns effective parameter policies more sample-efficiently than several baselines. Experiments in the real-world demonstrate our approach's ability to handle noise from perception and control and improve the robot's ability to solve two long-horizon mobile-manipulation tasks after a few hours of autonomous practice. Project website: http://ees.csail.mit.edu