# Feedback Control For Cassie With Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1803.05580 · HTML: https://mlsanity.com/paper/1803.05580/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2018.

## Built on by (most cited first, 30 of 45)

- [Real-World Humanoid Locomotion with Reinforcement Learning](https://mlsanity.com/paper/2303.03381/) (2023)
- [HumanPlus: Humanoid Shadowing and Imitation from Humans](https://mlsanity.com/paper/2406.10454/) (2024)
- [BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion](https://mlsanity.com/paper/2508.08241/) (2025)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control](https://mlsanity.com/paper/2401.16889/) (2024)
- [TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction](https://mlsanity.com/paper/2405.10315/) (2024)
- [Adapting Rapid Motor Adaptation for Bipedal Robots](https://mlsanity.com/paper/2205.15299/) (2022)
- [Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotion](https://mlsanity.com/paper/2308.12517/) (2023)
- [Robust and Versatile Bipedal Jumping Control through Reinforcement Learning](https://mlsanity.com/paper/2302.09450/) (2023)
- [Machine Learning for Microcontroller-Class Hardware: A Review](https://mlsanity.com/paper/2205.14550/) (2022)
- [A Survey on Physics Informed Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2309.01909/) (2023)
- [Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraints](https://mlsanity.com/paper/2301.10923/) (2023)
- [Learning Low-Frequency Motion Control for Robust and Dynamic Robot Locomotion](https://mlsanity.com/paper/2209.14887/) (2022)
- [OstrichRL: A Musculoskeletal Ostrich Simulation to Study Bio-mechanical Locomotion](https://mlsanity.com/paper/2112.06061/) (2021)
- [High-Performance Reinforcement Learning on Spot: Optimizing Simulation Parameters with Distributional Measures](https://mlsanity.com/paper/2504.17857/) (2025)
- [Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience](https://mlsanity.com/paper/2109.11767/) (2021)
- [Olaf: Bringing an Animated Character to Life in the Physical World](https://mlsanity.com/paper/2512.16705/) (2025)
- [Learning Bipedal Robot Locomotion from Human Movement](https://mlsanity.com/paper/2105.12277/) (2021)
- [Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey](https://mlsanity.com/paper/2404.17070/) (2024)
- [Learning Forward Dynamics Model and Informed Trajectory Sampler for Safe Quadruped Navigation](https://mlsanity.com/paper/2204.08647/) (2022)
- [Improved Policy Optimization for Online Imitation Learning](https://mlsanity.com/paper/2208.00088/) (2022)
- [Bridging Model-based Safety and Model-free Reinforcement Learning through System Identification of Low Dimensional Linear Models](https://mlsanity.com/paper/2205.05787/) (2022)
- [Infer and Adapt: Bipedal Locomotion Reward Learning from Demonstrations via Inverse Reinforcement Learning](https://mlsanity.com/paper/2309.16074/) (2023)
- [RL-augmented MPC Framework for Agile and Robust Bipedal Footstep Locomotion Planning and Control](https://mlsanity.com/paper/2407.17683/) (2024)
- [Optimizing Bipedal Locomotion for The 100m Dash With Comparison to Human Running](https://mlsanity.com/paper/2508.03070/) (2025)
- [Human-AI Shared Control via Policy Dissection](https://mlsanity.com/paper/2206.00152/) (2022)
- [Multi-Fidelity Reinforcement Learning for Time-Optimal Quadrotor Re-planning](https://mlsanity.com/paper/2403.08152/) (2024)
- [Learning Linear Policies for Robust Bipedal Locomotion on Terrains with Varying Slopes](https://mlsanity.com/paper/2104.01662/) (2021)
- [Boosting Exploration in Actor-Critic Algorithms by Incentivizing Plausible Novel States](https://mlsanity.com/paper/2210.00211/) (2022)
- [MPC-Inspired Reinforcement Learning for Verifiable Model-Free Control](https://mlsanity.com/paper/2312.05332/) (2023)

## Abstract (arXiv)

Bipedal locomotion skills are challenging to develop. Control strategies often use local linearization of the dynamics in conjunction with reduced-order abstractions to yield tractable solutions. In these model-based control strategies, the controller is often not fully aware of many details, including torque limits, joint limits, and other non-linearities that are necessarily excluded from the control computations for simplicity. Deep reinforcement learning (DRL) offers a promising model-free approach for controlling bipedal locomotion which can more fully exploit the dynamics. However, current results in the machine learning literature are often based on ad-hoc simulation models that are not based on corresponding hardware. Thus it remains unclear how well DRL will succeed on realizable bipedal robots. In this paper, we demonstrate the effectiveness of DRL using a realistic model of Cassie, a bipedal robot. By formulating a feedback control problem as finding the optimal policy for a Markov Decision Process, we are able to learn robust walking controllers that imitate a reference motion with DRL. Controllers for different walking speeds are learned by imitating simple time-scaled versions of the original reference motion. Controller robustness is demonstrated through several challenging tests, including sensory delay, walking blindly on irregular terrain and unexpected pushes at the pelvis. We also show we can interpolate between individual policies and that robustness can be improved with an interpolated policy.