# RMP2: A Structured Composable Policy Class for Robot Learning

arXiv: https://arxiv.org/abs/2103.05922 · HTML: https://mlsanity.com/paper/2103.05922/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 10 Mar 2021.

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [Reinforcement and Imitation Learning via Interactive No-Regret Learning](https://mlsanity.com/paper/1406.5979/) (2014)
- [Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions](https://mlsanity.com/paper/2004.07584/) (2020)
- [Riemannian Motion Policies](https://mlsanity.com/paper/1801.02854/) (2018)
- [ImitationFlow: Learning Deep Stable Stochastic Dynamic Systems by Normalizing Flows](https://mlsanity.com/paper/2010.13129/) (2020)
- [Euclideanizing Flows: Diffeomorphic Reduction for Learning Stable Dynamical Systems](https://mlsanity.com/paper/2005.13143/) (2020)
- [Real-time Perception meets Reactive Motion Generation](https://mlsanity.com/paper/1703.03512/) (2017)
- [RMPflow: A Computational Graph for Automatic Motion Policy Generation](https://mlsanity.com/paper/1811.07049/) (2018)
- [Representing Robot Task Plans as Robust Logical-Dynamical Systems](https://mlsanity.com/paper/1908.01896/) (2019)
- [Learning Vision-based Reactive Policies for Obstacle Avoidance](https://mlsanity.com/paper/2010.16298/) (2020)
- [RMPflow: A Geometric Framework for Generation of Multi-Task Motion Policies](https://mlsanity.com/paper/2007.14256/) (2020)
- [Joint Space Control via Deep Reinforcement Learning](https://mlsanity.com/paper/2011.06332/) (2020)
- [Neural Autonomous Navigation with Riemannian Motion Policy](https://mlsanity.com/paper/1904.01762/) (2019)
- [Riemannian Motion Policy Fusion through Learnable Lyapunov Function Reshaping](https://mlsanity.com/paper/1910.02646/) (2019)
- [Reinforcement Learning Experiments and Benchmark for Solving Robotic Reaching Tasks](https://mlsanity.com/paper/2011.05782/) (2020)
- [Learning Latent Space Dynamics for Tactile Servoing](https://mlsanity.com/paper/1811.03704/) (2018)
- [Multi-Objective Policy Generation for Multi-Robot Systems Using Riemannian Motion Policies](https://mlsanity.com/paper/1902.05177/) (2019)
- [Stable, Concurrent Controller Composition for Multi-Objective Robotic Tasks](https://mlsanity.com/paper/1903.12605/) (2019)
- [Towards Coordinated Robot Motions: End-to-End Learning of Motion Policies on Transform Trees](https://mlsanity.com/paper/2012.13457/) (2020)

## Built on by (most cited first, 8 of 8)

- [DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Training](https://mlsanity.com/paper/2305.12127/) (2023)
- [How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition](https://mlsanity.com/paper/2207.07730/) (2022)
- [GarmentLab: A Unified Simulation and Benchmark for Garment Manipulation](https://mlsanity.com/paper/2411.01200/) (2024)
- [Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning](https://mlsanity.com/paper/2105.04962/) (2021)
- [Hierarchical Policy Blending as Inference for Reactive Robot Control](https://mlsanity.com/paper/2210.07890/) (2022)
- [LeTO: Learning Constrained Visuomotor Policy with Differentiable Trajectory Optimization](https://mlsanity.com/paper/2401.17500/) (2024)
- [Imitation Learning via Simultaneous Optimization of Policies and Auxiliary Trajectories](https://mlsanity.com/paper/2105.03019/) (2021)
- [World-Task Factorization for Robot Learning](https://mlsanity.com/paper/2606.02027/) (2026)

## Abstract (arXiv)

We consider the problem of learning motion policies for acceleration-based robotics systems with a structured policy class specified by RMPflow. RMPflow is a multi-task control framework that has been successfully applied in many robotics problems. Using RMPflow as a structured policy class in learning has several benefits, such as sufficient expressiveness, the flexibility to inject different levels of prior knowledge as well as the ability to transfer policies between robots. However, implementing a system for end-to-end learning RMPflow policies faces several computational challenges. In this work, we re-examine the message passing algorithm of RMPflow and propose a more efficient alternate algorithm, called RMP2, that uses modern automatic differentiation tools (such as TensorFlow and PyTorch) to compute RMPflow policies. Our new design retains the strengths of RMPflow while bringing in advantages from automatic differentiation, including 1) easy programming interfaces to designing complex transformations; 2) support of general directed acyclic graph (DAG) transformation structures; 3) end-to-end differentiability for policy learning; 4) improved computational efficiency. Because of these features, RMP2 can be treated as a structured policy class for efficient robot learning which is suitable encoding domain knowledge. Our experiments show that using structured policy class given by RMP2 can improve policy performance and safety in reinforcement learning tasks for goal reaching in cluttered space.