# Combining model-predictive control and predictive reinforcement learning for stable quadrupedal robot locomotion

arXiv: https://arxiv.org/abs/2307.07752 · HTML: https://mlsanity.com/paper/2307.07752/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 15 Jul 2023.

## Builds on (most cited first, 14 of 14)

- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion](https://mlsanity.com/paper/2202.05481/) (2022)
- [Highly Dynamic Quadruped Locomotion via Whole-Body Impulse Control and Model Predictive Control](https://mlsanity.com/paper/1909.06586/) (2019)
- [Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot](https://mlsanity.com/paper/2003.03200/) (2020)
- [Optimal Cost Design for Model Predictive Control](https://mlsanity.com/paper/2104.11353/) (2021)
- [Reinforcement Learning of the Prediction Horizon in Model Predictive Control](https://mlsanity.com/paper/2102.11122/) (2021)
- [Optimization of the Model Predictive Control Meta-Parameters Through Reinforcement Learning](https://mlsanity.com/paper/2111.04146/) (2021)
- [Adaptive CLF-MPC With Application To Quadrupedal Robots](https://mlsanity.com/paper/2112.04536/) (2021)
- [Learning-based MPC from Big Data Using Reinforcement Learning](https://mlsanity.com/paper/2301.01667/) (2023)
- [An experimental study of two predictive reinforcement learning methods and comparison with model-predictive control](https://mlsanity.com/paper/2108.04857/) (2021)
- [Model predictive control of indoor microclimate: existing building stock comfort improvement](https://mlsanity.com/paper/1806.08999/) (2018)

## Abstract (arXiv)

Stable gait generation is a crucial problem for legged robot locomotion as this impacts other critical performance factors such as, e.g. mobility over an uneven terrain and power consumption. Gait generation stability results from the efficient control of the interaction between the legged robot's body and the environment where it moves. Here, we study how this can be achieved by a combination of model-predictive and predictive reinforcement learning controllers. Model-predictive control (MPC) is a well-established method that does not utilize any online learning (except for some adaptive variations) as it provides a convenient interface for state constraints management. Reinforcement learning (RL), in contrast, relies on adaptation based on pure experience. In its bare-bone variants, RL is not always suitable for robots due to their high complexity and expensive simulation/experimentation. In this work, we combine both control methods to address the quadrupedal robot stable gate generation problem. The hybrid approach that we develop and apply uses a cost roll-out algorithm with a tail cost in the form of a Q-function modeled by a neural network; this allows to alleviate the computational complexity, which grows exponentially with the prediction horizon in a purely MPC approach. We demonstrate that our RL gait controller achieves stable locomotion at short horizons, where a nominal MP controller fails. Further, our controller is capable of live operation, meaning that it does not require previous training. Our results suggest that the hybridization of MPC with RL, as presented here, is beneficial to achieve a good balance between online control capabilities and computational complexity.