# RL + Model-based Control: Using On-demand Optimal Control to Learn Versatile Legged Locomotion

arXiv: https://arxiv.org/abs/2305.17842 · HTML: https://mlsanity.com/paper/2305.17842/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [Rapid Locomotion via Reinforcement Learning](https://mlsanity.com/paper/2205.02824/) (2022)
- [Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior](https://mlsanity.com/paper/2212.03238/) (2022)
- [Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions](https://mlsanity.com/paper/2203.15103/) (2022)
- [RLOC: Terrain-Aware Legged Locomotion using Reinforcement Learning and Optimal Control](https://mlsanity.com/paper/2012.03094/) (2020)
- [Highly Dynamic Quadruped Locomotion via Whole-Body Impulse Control and Model Predictive Control](https://mlsanity.com/paper/1909.06586/) (2019)
- [Policies Modulating Trajectory Generators](https://mlsanity.com/paper/1910.02812/) (2019)
- [Perceptive Locomotion through Nonlinear Model Predictive Control](https://mlsanity.com/paper/2208.08373/) (2022)
- [OPT-Mimic: Imitation of Optimized Trajectories for Dynamic Quadruped Behaviors](https://mlsanity.com/paper/2210.01247/) (2022)
- [Optimization-Based Control for Dynamic Legged Robots](https://mlsanity.com/paper/2211.11644/) (2022)
- [Learning Free Gait Transition for Quadruped Robots via Phase-Guided Controller](https://mlsanity.com/paper/2201.00206/) (2022)
- [Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions](https://mlsanity.com/paper/2209.07899/) (2022)
- [TAMOLS: Terrain-Aware Motion Optimization for Legged Systems](https://mlsanity.com/paper/2206.14049/) (2022)
- [Motion Planning for Quadrupedal Locomotion: Coupled Planning, Terrain Mapping and Whole-Body Control](https://mlsanity.com/paper/2003.05481/) (2020)
- [Kinodynamic Motion Planning for Multi-Legged Robot Jumping via Mixed-Integer Convex Program](https://mlsanity.com/paper/2011.01809/) (2020)
- [Online Trajectory Optimization for Dynamic Aerial Motions of a Quadruped Robot](https://mlsanity.com/paper/2110.06330/) (2021)
- [Nonlinear Model Predictive Control for Quadrupedal Locomotion Using Second-Order Sensitivity Analysis](https://mlsanity.com/paper/2207.10465/) (2022)
- [Reinforcement Learning for Legged Robots: Motion Imitation from Model-Based Optimal Control](https://mlsanity.com/paper/2305.10989/) (2023)
- [Optimal control via second order sensitivity analysis](https://mlsanity.com/paper/1905.08534/) (2019)

## Built on by (most cited first, 26 of 26)

- [Robot Parkour Learning](https://mlsanity.com/paper/2309.05665/) (2023)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewards](https://mlsanity.com/paper/2407.11562/) (2024)
- [Design and Control of a Bipedal Robotic Character](https://mlsanity.com/paper/2501.05204/) (2025)
- [Guided Reinforcement Learning for Robust Multi-Contact Loco-Manipulation](https://mlsanity.com/paper/2410.13817/) (2024)
- [Tuning Legged Locomotion Controllers via Safe Bayesian Optimization](https://mlsanity.com/paper/2306.07092/) (2023)
- [Learning Diverse Skills for Local Navigation under Multi-constraint Optimality](https://mlsanity.com/paper/2310.02440/) (2023)
- [Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Models](https://mlsanity.com/paper/2410.00231/) (2024)
- [Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey](https://mlsanity.com/paper/2404.17070/) (2024)
- [A roadmap for AI in robotics](https://mlsanity.com/paper/2507.19975/) (2025)
- [Achieving Stable High-Speed Locomotion for Humanoid Robots with Deep Reinforcement Learning](https://mlsanity.com/paper/2409.16611/) (2024)
- [RL-augmented MPC Framework for Agile and Robust Bipedal Footstep Locomotion Planning and Control](https://mlsanity.com/paper/2407.17683/) (2024)
- [DeepTransition: Viability Leads to the Emergence of Gait Transitions in Learning Anticipatory Quadrupedal Locomotion Skills](https://mlsanity.com/paper/2306.07419/) (2023)
- [Flip Stunts on Bicycle Robots using Iterative Motion Imitation](https://mlsanity.com/paper/2603.27944/) (2026)
- [MetaLoco: Universal Quadrupedal Locomotion with Meta-Reinforcement Learning and Motion Imitation](https://mlsanity.com/paper/2407.17502/) (2024)
- [Reinforcement Learning Compensated Model Predictive Control for Off-road Driving on Unknown Deformable Terrain](https://mlsanity.com/paper/2408.09253/) (2024)
- [Integrating Trajectory Optimization and Reinforcement Learning for Quadrupedal Jumping with Terrain-Adaptive Landing](https://mlsanity.com/paper/2509.12776/) (2025)
- [Agile perceptive multi-skill locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2607.13579/) (2026)
- [Learning to Navigate from Scratch using World Models and Curiosity: the Good, the Bad, and the Ugly](https://mlsanity.com/paper/2308.15852/) (2023)
- [Learning Generic and Dynamic Locomotion of Humanoids Across Discrete Terrains](https://mlsanity.com/paper/2405.17227/) (2024)
- [Whole-Body Constrained Learning for Legged Locomotion via Hierarchical Optimization](https://mlsanity.com/paper/2506.05115/) (2025)
- [Dynamic Adaptive Legged Locomotion Policy via Decoupling Reaction Force Control and Gait Control](https://mlsanity.com/paper/2509.13737/) (2025)
- [HybridMimic: Hybrid RL-Centroidal Control for Humanoid Motion Mimicking](https://mlsanity.com/paper/2603.06775/) (2026)
- [RL-Augmented MPC for Non-Gaited Legged and Hybrid Locomotion](https://mlsanity.com/paper/2603.10878/) (2026)
- [Bicycle Acrobatics with Reinforcement Learning](https://mlsanity.com/paper/2608.00880/) (2026)
- [Guided Riemannian Optimization (GuRO): Bridging Model Predictive Control and Decision Transformers](https://mlsanity.com/paper/2608.23204/) (2026)

## Abstract (arXiv)

This paper presents a control framework that combines model-based optimal control and reinforcement learning (RL) to achieve versatile and robust legged locomotion. Our approach enhances the RL training process by incorporating on-demand reference motions generated through finite-horizon optimal control, covering a broad range of velocities and gaits. These reference motions serve as targets for the RL policy to imitate, leading to the development of robust control policies that can be learned with reliability. Furthermore, by utilizing realistic simulation data that captures whole-body dynamics, RL effectively overcomes the inherent limitations in reference motions imposed by modeling simplifications. We validate the robustness and controllability of the RL training process within our framework through a series of experiments. In these experiments, our method showcases its capability to generalize reference motions and effectively handle more complex locomotion tasks that may pose challenges for the simplified model, thanks to RL's flexibility. Additionally, our framework effortlessly supports the training of control policies for robots with diverse dimensions, eliminating the necessity for robot-specific adjustments in the reward function and hyperparameters.