# On the Emergence of Whole-body Strategies from Humanoid Robot Push-recovery Learning

arXiv: https://arxiv.org/abs/2104.14534 · HTML: https://mlsanity.com/paper/2104.14534/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 45 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [An empirical investigation of the challenges of real-world reinforcement learning](https://mlsanity.com/paper/2003.11881/) (2020)
- [Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model](https://mlsanity.com/paper/1610.03518/) (2016)
- [Assessing Transferability from Simulation to Reality for Reinforcement Learning](https://mlsanity.com/paper/1907.04685/) (2019)
- [Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie](https://mlsanity.com/paper/1903.09537/) (2019)
- [Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learning](https://mlsanity.com/paper/1911.01715/) (2019)
- [Learning natural locomotion behaviors for humanoid robots using human knowledge](https://mlsanity.com/paper/2005.10195/) (2020)
- [Policy Search in Continuous Action Domains: an Overview](https://mlsanity.com/paper/1803.04706/) (2018)
- [Hybrid Zero Dynamics Inspired Feedback Control Policy Design for 3D Bipedal Locomotion using Reinforcement Learning](https://mlsanity.com/paper/1910.01748/) (2019)
- [Learning Whole-body Motor Skills for Humanoids](https://mlsanity.com/paper/2002.02991/) (2020)
- [Emergence of Human-comparable Balancing Behaviors by Deep Reinforcement Learning](https://mlsanity.com/paper/1809.02074/) (2018)
- [A Control Architecture with Online Predictive Planning for Position and Torque Controlled Walking of Humanoid Robots](https://mlsanity.com/paper/1807.05395/) (2018)
- [A Benchmarking of DCM Based Architectures for Position and Velocity Controlled Walking of Humanoid Robots](https://mlsanity.com/paper/1809.02167/) (2018)
- [Stability Analysis and Design of Momentum-based Controllers for Humanoid Robots](https://mlsanity.com/paper/1603.04178/) (2016)
- [Torque-Controlled Stepping-Strategy Push Recovery: Design and Implementation on the iCub Humanoid Robot](https://mlsanity.com/paper/1705.10579/) (2017)

## Built on by (most cited first, 3 of 3)

- [Enhancing Model-Based Step Adaptation for Push Recovery through Reinforcement Learning of Step Timing and Region](https://mlsanity.com/paper/2411.01000/) (2024)
- [Constrained Reinforcement Learning for Unstable Point-Feet Bipedal Locomotion Applied to the Bolt Robot](https://mlsanity.com/paper/2508.02194/) (2025)
- [Learning Impact-Rich Rotational Maneuvers via Centroidal Velocity Rewards and Sim-to-Real Techniques: A One-Leg Hopper Flip Case Study](https://mlsanity.com/paper/2505.12222/) (2025)

## Abstract (arXiv)

Balancing and push-recovery are essential capabilities enabling humanoid robots to solve complex locomotion tasks. In this context, classical control systems tend to be based on simplified physical models and hard-coded strategies. Although successful in specific scenarios, this approach requires demanding tuning of parameters and switching logic between specifically-designed controllers for handling more general perturbations. We apply model-free Deep Reinforcement Learning for training a general and robust humanoid push-recovery policy in a simulation environment. Our method targets high-dimensional whole-body humanoid control and is validated on the iCub humanoid. Reward components incorporating expert knowledge on humanoid control enable fast learning of several robust behaviors by the same policy, spanning the entire body. We validate our method with extensive quantitative analyses in simulation, including out-of-sample tasks which demonstrate policy robustness and generalization, both key requirements towards real-world robot deployment.