# Model-free Reinforcement Learning for Robust Locomotion using Demonstrations from Trajectory Optimization

arXiv: https://arxiv.org/abs/2107.06629 · HTML: https://mlsanity.com/paper/2107.06629/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2105.08328/) (2021)
- [Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotion](https://mlsanity.com/paper/2011.02404/) (2020)
- [RLOC: Terrain-Aware Legged Locomotion using Reinforcement Learning and Optimal Control](https://mlsanity.com/paper/2012.03094/) (2020)
- [Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?](https://mlsanity.com/paper/1611.01055/) (2016)
- [GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model](https://mlsanity.com/paper/2104.09771/) (2021)
- [An Open Torque-Controlled Modular Robot Architecture for Legged Locomotion Research](https://mlsanity.com/paper/1910.00093/) (2019)
- [Learning Variable Impedance Control for Contact Sensitive Tasks](https://mlsanity.com/paper/1907.07500/) (2019)
- [Efficient Multi-Contact Pattern Generation with Sequential Convex Approximations of the Centroidal Dynamics](https://mlsanity.com/paper/2010.01215/) (2020)
- [Robust Trajectory Optimization over Uncertain Terrain with Stochastic Complementarity](https://mlsanity.com/paper/2009.12409/) (2020)
- [Stabilization of Complementarity Systems via Contact-Aware Controllers](https://mlsanity.com/paper/2008.02104/) (2020)
- [Impedance Optimization for Uncertain Contact Interactions Through Risk Sensitive Optimal Control](https://mlsanity.com/paper/2011.04684/) (2020)

## Built on by (most cited first, 19 of 19)

- [Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control](https://mlsanity.com/paper/2401.16889/) (2024)
- [DTC: Deep Tracking Control](https://mlsanity.com/paper/2309.15462/) (2023)
- [Robust and Versatile Bipedal Jumping Control through Reinforcement Learning](https://mlsanity.com/paper/2302.09450/) (2023)
- [Learning and Adapting Agile Locomotion Skills by Transferring Experience](https://mlsanity.com/paper/2304.09834/) (2023)
- [Creating a Dynamic Quadrupedal Robotic Goalkeeper with Reinforcement Learning](https://mlsanity.com/paper/2210.04435/) (2022)
- [SoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experience](https://mlsanity.com/paper/2409.13678/) (2024)
- [ZEST: Zero-shot Embodied Skill Transfer for Athletic Robot Control](https://mlsanity.com/paper/2602.00401/) (2026)
- [Guided Reinforcement Learning for Robust Multi-Contact Loco-Manipulation](https://mlsanity.com/paper/2410.13817/) (2024)
- [Parameter-Adaptive Approximate MPC: Tuning Neural-Network Controllers without Retraining](https://mlsanity.com/paper/2404.05835/) (2024)
- [Unsupervised Skill Discovery as Exploration for Learning Agile Locomotion](https://mlsanity.com/paper/2508.08982/) (2025)
- [Guided Reinforcement Learning for Omnidirectional 3D Jumping in Quadruped Robots](https://mlsanity.com/paper/2507.16481/) (2025)
- [Just Round: Quantized Observation Spaces Enable Memory Efficient Learning of Dynamic Locomotion](https://mlsanity.com/paper/2210.08065/) (2022)
- [Characterization of Human Balance through a Reinforcement Learning-based Muscle Controller](https://mlsanity.com/paper/2308.04462/) (2023)
- [Efficient Reinforcement Learning for Jumping Monopods](https://mlsanity.com/paper/2309.07038/) (2023)
- [SLowRL: Safe Low-Rank Adaptation Reinforcement Learning for Locomotion](https://mlsanity.com/paper/2603.17092/) (2026)
- [MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins](https://mlsanity.com/paper/2606.26392/) (2026)
- [ADP: Adversarial Dynamics Priors for Physically Grounded Humanoid Locomotion](https://mlsanity.com/paper/2607.03454/) (2026)
- [CMR: Contractive Mapping Embeddings for Robust Humanoid Locomotion on Unstructured Terrains](https://mlsanity.com/paper/2602.03511/) (2026)
- [SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions](https://mlsanity.com/paper/2607.08925/) (2026)

## Abstract (arXiv)

We present a general, two-stage reinforcement learning approach to create robust policies that can be deployed on real robots without any additional training using a single demonstration generated by trajectory optimization. The demonstration is used in the first stage as a starting point to facilitate initial exploration. In the second stage, the relevant task reward is optimized directly and a policy robust to environment uncertainties is computed. We demonstrate and examine in detail the performance and robustness of our approach on highly dynamic hopping and bounding tasks on a quadruped robot.