# Learning Setup Policies: Reliable Transition Between Locomotion Behaviours

arXiv: https://arxiv.org/abs/2101.09391 · HTML: https://mlsanity.com/paper/2101.09391/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 23 Jan 2021.

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies](https://mlsanity.com/paper/1905.09808/) (2019)
- [Meta Learning Shared Hierarchies](https://mlsanity.com/paper/1710.09767/) (2017)
- [Multi-expert learning of adaptive legged locomotion](https://mlsanity.com/paper/2012.05810/) (2020)
- [ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills](https://mlsanity.com/paper/2005.04323/) (2020)
- [Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning](https://mlsanity.com/paper/1901.07517/) (2019)
- [Hierarchical visuomotor control of humanoids](https://mlsanity.com/paper/1811.09656/) (2018)
- [Hierarchical Style-based Networks for Motion Synthesis](https://mlsanity.com/paper/2008.10162/) (2020)
- [Guided Curriculum Learning for Walking Over Complex Terrain](https://mlsanity.com/paper/2010.03848/) (2020)
- [Recurrent Deterministic Policy Gradient Method for Bipedal Locomotion on Rough Terrain Challenge](https://mlsanity.com/paper/1710.02896/) (2017)
- [Learning When to Switch: Composing Controllers to Traverse a Sequence of Terrain Artifacts](https://mlsanity.com/paper/2011.00440/) (2020)
- [Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning](https://mlsanity.com/paper/2011.08345/) (2020)

## Built on by (most cited first, 2 of 2)

- [Passing Through Narrow Gaps with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.03991/) (2021)
- [Skill Composition for Legged Robot Reinforcement Learning](https://mlsanity.com/paper/2609.14647/) (2026)

## Abstract (arXiv)

Dynamic platforms that operate over many unique terrain conditions typically require many behaviours. To transition safely, there must be an overlap of states between adjacent controllers. We develop a novel method for training setup policies that bridge the trajectories between pre-trained Deep Reinforcement Learning (DRL) policies. We demonstrate our method with a simulated biped traversing a difficult jump terrain, where a single policy fails to learn the task, and switching between pre-trained policies without setup policies also fails. We perform an ablation of key components of our system, and show that our method outperforms others that learn transition policies. We demonstrate our method with several difficult and diverse terrain types, and show that we can use setup policies as part of a modular control suite to successfully traverse a sequence of complex terrains. We show that using setup policies improves the success rate for traversing a single difficult jump terrain (from 51.3% success rate with the best comparative method to 82.2%), and traversing a random sequence of difficult obstacles (from 1.9% without setup policies to 71.2%).