# Learning Memory-Based Control for Human-Scale Bipedal Locomotion

arXiv: https://arxiv.org/abs/2006.02402 · HTML: https://mlsanity.com/paper/2006.02402/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2020.

## Built on by (most cited first, 28 of 28)

- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control](https://mlsanity.com/paper/2401.16889/) (2024)
- [Adapting Rapid Motor Adaptation for Bipedal Robots](https://mlsanity.com/paper/2205.15299/) (2022)
- [Robust and Versatile Bipedal Jumping Control through Reinforcement Learning](https://mlsanity.com/paper/2302.09450/) (2023)
- [Learning Free Gait Transition for Quadruped Robots via Phase-Guided Controller](https://mlsanity.com/paper/2201.00206/) (2022)
- [CTS: Concurrent Teacher-Student Reinforcement Learning for Legged Locomotion](https://mlsanity.com/paper/2405.10830/) (2024)
- [Robust High-speed Running for Quadruped Robots via Deep Reinforcement Learning](https://mlsanity.com/paper/2103.06484/) (2021)
- [Learning Bipedal Walking for Humanoids with Current Feedback](https://mlsanity.com/paper/2303.03724/) (2023)
- [Meta-Reinforcement Learning for the Tuning of PI Controllers: An Offline Approach](https://mlsanity.com/paper/2203.09661/) (2022)
- [Distillation-PPO: A Novel Two-Stage Reinforcement Learning Framework for Humanoid Robot Perceptive Locomotion](https://mlsanity.com/paper/2503.08299/) (2025)
- [Video2Reward: Generating Reward Function from Videos for Legged Robot Behavior Learning](https://mlsanity.com/paper/2412.05515/) (2024)
- [Residual Physics Learning and System Identification for Sim-to-real Transfer of Policies on Buoyancy Assisted Legged Robots](https://mlsanity.com/paper/2303.09597/) (2023)
- [Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey](https://mlsanity.com/paper/2404.17070/) (2024)
- [Reinforcement Learning Within the Classical Robotics Stack: A Case Study in Robot Soccer](https://mlsanity.com/paper/2412.09417/) (2024)
- [RL-augmented MPC Framework for Agile and Robust Bipedal Footstep Locomotion Planning and Control](https://mlsanity.com/paper/2407.17683/) (2024)
- [TAR: Teacher-Aligned Representations via Contrastive Learning for Quadrupedal Locomotion](https://mlsanity.com/paper/2503.20839/) (2025)
- [Optimizing Bipedal Locomotion for The 100m Dash With Comparison to Human Running](https://mlsanity.com/paper/2508.03070/) (2025)
- [Learning Decentralized Multi-Biped Control for Payload Transport](https://mlsanity.com/paper/2406.17279/) (2024)
- [Learning Linear Policies for Robust Bipedal Locomotion on Terrains with Varying Slopes](https://mlsanity.com/paper/2104.01662/) (2021)
- [Robust Humanoid Walking on Compliant and Uneven Terrain with Deep Reinforcement Learning](https://mlsanity.com/paper/2504.13619/) (2025)
- [LIPM-Guided Reinforcement Learning for Stable and Perceptive Locomotion in Bipedal Robots](https://mlsanity.com/paper/2509.09106/) (2025)
- [From Data-Fitting to Discovery: Interpreting the Neural Dynamics of Motor Control through Reinforcement Learning](https://mlsanity.com/paper/2305.11107/) (2023)
- [MetaLoco: Universal Quadrupedal Locomotion with Meta-Reinforcement Learning and Motion Imitation](https://mlsanity.com/paper/2407.17502/) (2024)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Bipedal Locomotion](https://mlsanity.com/paper/2511.06465/) (2025)
- [A Population-Level Analysis of Neural Dynamics in Robust Legged Robots](https://mlsanity.com/paper/2306.15793/) (2023)
- [Learning to Dock: A Simulation-based Study on Closing the Sim2Real Gap in Autonomous Underwater Docking](https://mlsanity.com/paper/2506.17823/) (2025)
- [No More Marching: Learning Humanoid Locomotion for Short-Range SE(2) Targets](https://mlsanity.com/paper/2508.14098/) (2025)

## Abstract (arXiv)

Controlling a non-statically stable biped is a difficult problem largely due to the complex hybrid dynamics involved. Recent work has demonstrated the effectiveness of reinforcement learning (RL) for simulation-based training of neural network controllers that successfully transfer to real bipeds. The existing work, however, has primarily used simple memoryless network architectures, even though more sophisticated architectures, such as those including memory, often yield superior performance in other RL domains. In this work, we consider recurrent neural networks (RNNs) for sim-to-real biped locomotion, allowing for policies that learn to use internal memory to model important physical properties. We show that while RNNs are able to significantly outperform memoryless policies in simulation, they do not exhibit superior behavior on the real biped due to overfitting to the simulation physics unless trained using dynamics randomization to prevent overfitting; this leads to consistently better sim-to-real transfer. We also show that RNNs could use their learned memory states to perform online system identification by encoding parameters of the dynamics into memory.