# MOSAIC: Skill-Centric Manipulation Planning with Physics Simulation

arXiv: https://arxiv.org/abs/2504.16738 · HTML: https://mlsanity.com/paper/2504.16738/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Builds on (most cited first, 26 of 26)

- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills](https://mlsanity.com/paper/2302.04659/) (2023)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models](https://mlsanity.com/paper/2308.01557/) (2023)
- [Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation](https://mlsanity.com/paper/2403.03949/) (2024)
- [Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models](https://mlsanity.com/paper/2401.03360/) (2023)
- [Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning](https://mlsanity.com/paper/1708.02190/) (2017)
- [Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks](https://mlsanity.com/paper/2110.03655/) (2021)
- [MuJoCo Playground](https://mlsanity.com/paper/2502.08844/) (2025)
- [Learning Neuro-Symbolic Skills for Bilevel Planning](https://mlsanity.com/paper/2206.10680/) (2022)
- [Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics](https://mlsanity.com/paper/2501.10100/) (2025)
- [Learning compositional models of robot skills for task and motion planning](https://mlsanity.com/paper/2006.06444/) (2020)
- [Deep Affordance Foresight: Planning Through What Can Be Done in the Future](https://mlsanity.com/paper/2011.08424/) (2020)
- [Reset-Free Lifelong Learning with Skill-Space Planning](https://mlsanity.com/paper/2012.03548/) (2020)
- [Discovery of Options via Meta-Learned Subgoals](https://mlsanity.com/paper/2102.06741/) (2021)
- [DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observability](https://mlsanity.com/paper/2306.13196/) (2023)
- [Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation](https://mlsanity.com/paper/2109.08771/) (2021)
- [Fast Contact-Implicit Model-Predictive Control](https://mlsanity.com/paper/2107.05616/) (2021)
- [Generative Factor Chaining: Coordinated Manipulation with Diffusion-based Factor Graph](https://mlsanity.com/paper/2409.16275/) (2024)
- [Manipulation Planning Among Movable Obstacles Using Physics-Based Adaptive Motion Primitives](https://mlsanity.com/paper/2102.04324/) (2021)
- [Symbolic State Estimation with Predicates for Contact-Rich Manipulation Tasks](https://mlsanity.com/paper/2203.02468/) (2022)
- [Planning with Selective Physics-based Simulation for Manipulation Among Movable Objects](https://mlsanity.com/paper/2003.06743/) (2020)
- [Planning for Manipulation among Movable Objects: Deciding Which Objects Go Where, in What Order, and How](https://mlsanity.com/paper/2303.13385/) (2023)
- [Roadmaps with Gaps over Controllers: Achieving Efficiency in Planning under Dynamics](https://mlsanity.com/paper/2310.03239/) (2023)

## Built on by (most cited first, 2 of 2)

- [Human Centric General Physical Intelligence for Agile Manufacturing Automation](https://mlsanity.com/paper/2508.11960/) (2025)
- [From Code to Action: Hierarchical Learning of Diffusion-VLM Policies](https://mlsanity.com/paper/2509.24917/) (2025)

## Abstract (arXiv)

Planning long-horizon manipulation motions using a set of predefined skills is a central challenge in robotics; solving it efficiently could enable general-purpose robots to tackle novel tasks by flexibly composing generic skills. Solutions to this problem lie in an infinitely vast space of parameterized skill sequences -- a space where common incremental methods struggle to find sequences that have non-obvious intermediate steps. Some approaches reason over lower-dimensional, symbolic spaces, which are more tractable to explore but may be brittle and are laborious to construct. In this work, we introduce MOSAIC, a skill-centric, multi-directional planning approach that targets these challenges by reasoning about which skills to employ and where they are most likely to succeed, by utilizing physics simulation to estimate skill execution outcomes. Specifically, MOSAIC employs two complementary skill families: Generators, which identify ``islands of competence'' where skills are demonstrably effective, and Connectors, which link these skill-trajectories by solving boundary value problems. By focusing planning efforts on regions of high competence, MOSAIC efficiently discovers physically-grounded solutions. We demonstrate its efficacy on complex long-horizon problems in both simulation and the real world, using a diverse set of skills including generative diffusion models, motion planning algorithms, and manipulation-specific models. Visit skill-mosaic.github.io for demonstrations and examples.