# Hierarchical Policy Blending As Optimal Transport

arXiv: https://arxiv.org/abs/2212.01938 · HTML: https://mlsanity.com/paper/2212.01938/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 69 references that are in the map.
- First posted on arXiv 4 Dec 2022.

## Builds on (most cited first, 30 of 31)

- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration](https://mlsanity.com/paper/1705.09634/) (2017)
- [Learning with a Wasserstein Loss](https://mlsanity.com/paper/1506.05439/) (2015)
- [Watch and Match: Supercharging Imitation with Regularized Optimal Transport](https://mlsanity.com/paper/2206.15469/) (2022)
- [Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors](https://mlsanity.com/paper/2006.13205/) (2020)
- [On Unbalanced Optimal Transport: An Analysis of Sinkhorn Algorithm](https://mlsanity.com/paper/2002.03293/) (2020)
- [Neural Dynamic Policies for End-to-End Sensorimotor Learning](https://mlsanity.com/paper/2012.02788/) (2020)
- [Continuous-Time Gaussian Process Motion Planning via Probabilistic Inference](https://mlsanity.com/paper/1707.07383/) (2017)
- [ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation](https://mlsanity.com/paper/2008.07792/) (2020)
- [STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulation](https://mlsanity.com/paper/2104.13542/) (2021)
- [Riemannian Motion Policies](https://mlsanity.com/paper/1801.02854/) (2018)
- [Autonomous Reinforcement Learning via Subgoal Curricula](https://mlsanity.com/paper/2107.12931/) (2021)
- [Specializing Versatile Skill Libraries using Local Mixture of Experts](https://mlsanity.com/paper/2112.04216/) (2021)
- [Robot Learning of Mobile Manipulation with Reachability Behavior Priors](https://mlsanity.com/paper/2203.04051/) (2022)
- [RMPflow: A Computational Graph for Automatic Motion Policy Generation](https://mlsanity.com/paper/1811.07049/) (2018)
- [Learning Implicit Priors for Motion Optimization](https://mlsanity.com/paper/2204.05369/) (2022)
- [Continuous Action Reinforcement Learning from a Mixture of Interpretable Experts](https://mlsanity.com/paper/2006.05911/) (2020)
- [Hierarchical Neural Dynamic Policies](https://mlsanity.com/paper/2107.05627/) (2021)
- [Safe Reinforcement Learning of Dynamic High-Dimensional Robotic Tasks: Navigation, Manipulation, Interaction](https://mlsanity.com/paper/2209.13308/) (2022)
- [Regularized Deep Signed Distance Fields for Reactive Motion Generation](https://mlsanity.com/paper/2203.04739/) (2022)
- [Merging Position and Orientation Motion Primitives](https://mlsanity.com/paper/2003.11507/) (2020)
- [Hierarchical Policy Blending as Inference for Reactive Robot Control](https://mlsanity.com/paper/2210.07890/) (2022)
- [RMPflow: A Geometric Framework for Generation of Multi-Task Motion Policies](https://mlsanity.com/paper/2007.14256/) (2020)
- [Geometric Fabrics for the Acceleration-based Design of Robotic Motion](https://mlsanity.com/paper/2010.14750/) (2020)
- [Optimal Transport-based Coverage Control for Swarm Robot Systems: Generalization of the Voronoi Tessellation-based Method](https://mlsanity.com/paper/2011.08337/) (2020)
- [Learning to Sequence and Blend Robot Skills via Differentiable Optimization](https://mlsanity.com/paper/2206.00559/) (2022)
- [Adaptive Gaussian Process based Stochastic Trajectory Optimization for Motion Planning](https://mlsanity.com/paper/2112.14953/) (2021)
- [Fast Obstacle Avoidance Based on Real-Time Sensing](https://mlsanity.com/paper/2205.04928/) (2022)
- [Actor-Critic based Improper Reinforcement Learning](https://mlsanity.com/paper/2207.09090/) (2022)

## Built on by (most cited first, 4 of 4)

- [Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models](https://mlsanity.com/paper/2308.01557/) (2023)
- [Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learning](https://mlsanity.com/paper/2412.12953/) (2024)
- [Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts](https://mlsanity.com/paper/2403.06966/) (2024)
- [Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Model](https://mlsanity.com/paper/2407.04489/) (2024)

## Abstract (arXiv)

We present hierarchical policy blending as optimal transport (HiPBOT). HiPBOT hierarchically adjusts the weights of low-level reactive expert policies of different agents by adding a look-ahead planning layer on the parameter space. The high-level planner renders policy blending as unbalanced optimal transport consolidating the scaling of the underlying Riemannian motion policies. As a result, HiPBOT effectively decides the priorities between expert policies and agents, ensuring the task's success and guaranteeing safety. Experimental results in several application scenarios, from low-dimensional navigation to high-dimensional whole-body control, show the efficacy and efficiency of HiPBOT. Our method outperforms state-of-the-art baselines -- either adopting probabilistic inference or defining a tree structure of experts -- paving the way for new applications of optimal transport to robot control. More material at https://sites.google.com/view/hipobot