# PM-FSM: Policies Modulating Finite State Machine for Robust Quadrupedal Locomotion

arXiv: https://arxiv.org/abs/2109.12696 · HTML: https://mlsanity.com/paper/2109.12696/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 26 Sep 2021.

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [Learning to Walk via Deep Reinforcement Learning](https://mlsanity.com/paper/1812.11103/) (2018)
- [RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real](https://mlsanity.com/paper/2006.09001/) (2020)
- [Simple random search provides a competitive approach to reinforcement learning](https://mlsanity.com/paper/1803.07055/) (2018)
- [Data Efficient Reinforcement Learning for Legged Robots](https://mlsanity.com/paper/1907.03613/) (2019)
- [Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotion](https://mlsanity.com/paper/2011.02404/) (2020)
- [DeepGait: Planning and Control of Quadrupedal Gaits using Deep Reinforcement Learning](https://mlsanity.com/paper/1909.08399/) (2019)
- [Policies Modulating Trajectory Generators](https://mlsanity.com/paper/1910.02812/) (2019)
- [Learning Fast Adaptation with Meta Strategy Optimization](https://mlsanity.com/paper/1909.12995/) (2019)
- [GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model](https://mlsanity.com/paper/2104.09771/) (2021)

## Built on by (most cited first, 1 of 1)

- [HierKick: Hierarchical Reinforcement Learning for Vision-Guided Soccer Robot Control](https://mlsanity.com/paper/2603.00948/) (2026)

## Abstract (arXiv)

Deep reinforcement learning (deep RL) has emerged as an effective tool for developing controllers for legged robots. However, vanilla deep RL often requires a tremendous amount of training samples and is not feasible for achieving robust behaviors. Instead, researchers have investigated a novel policy architecture by incorporating human experts' knowledge, such as Policies Modulating Trajectory Generators (PMTG). This architecture builds a recurrent control loop by combining a parametric trajectory generator (TG) and a feedback policy network to achieve more robust behaviors. To take advantage of human experts' knowledge but eliminate time-consuming interactive teaching, researchers have investigated a novel architecture, Policies Modulating Trajectory Generators (PMTG), which builds a recurrent control loop by combining a parametric trajectory generator (TG) and a feedback policy network to achieve more robust behaviors using intuitive prior knowledge. In this work, we propose Policies Modulating Finite State Machine (PM-FSM) by replacing TGs with contact-aware finite state machines (FSM), which offer more flexible control of each leg. Compared with the TGs, FSMs offer high-level management on each leg motion generator and enable a flexible state arrangement, which makes the learned behavior less vulnerable to unseen perturbations or challenging terrains. This invention offers an explicit notion of contact events to the policy to negotiate unexpected perturbations. We demonstrated that the proposed architecture could achieve more robust behaviors in various scenarios, such as challenging terrains or external perturbations, on both simulated and real robots. The supplemental video can be found at: https://youtu.be/78cboMqTkJQ.