# Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie

arXiv: https://arxiv.org/abs/1903.09537 · HTML: https://mlsanity.com/paper/1903.09537/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2019.

## Built on by (most cited first, 14 of 14)

- [Expressive Whole-Body Control for Humanoid Robots](https://mlsanity.com/paper/2402.16796/) (2024)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning](https://mlsanity.com/paper/2405.02425/) (2024)
- [Learning Robust, Agile, Natural Legged Locomotion Skills in the Wild](https://mlsanity.com/paper/2304.10888/) (2023)
- [Learning Bipedal Robot Locomotion from Human Movement](https://mlsanity.com/paper/2105.12277/) (2021)
- [Learning Perceptual Locomotion on Uneven Terrains using Sparse Visual Observations](https://mlsanity.com/paper/2109.14026/) (2021)
- [Imitation and Adaptation Based on Consistency: A Quadruped Robot Imitates Animals from Videos Using Deep Reinforcement Learning](https://mlsanity.com/paper/2203.05973/) (2022)
- [Learning Linear Policies for Robust Bipedal Locomotion on Terrains with Varying Slopes](https://mlsanity.com/paper/2104.01662/) (2021)
- [On the Emergence of Whole-body Strategies from Humanoid Robot Push-recovery Learning](https://mlsanity.com/paper/2104.14534/) (2021)
- [A Simple Approach to Continual Learning by Transferring Skill Parameters](https://mlsanity.com/paper/2110.10255/) (2021)
- [Pedestrian crossing decisions can be explained by bounded optimal decision-making under noisy visual perception](https://mlsanity.com/paper/2402.04370/) (2024)
- [Value Summation: A Novel Scoring Function for MPC-based Model-based Reinforcement Learning](https://mlsanity.com/paper/2209.08169/) (2022)
- [Achieving Precise and Reliable Locomotion with Differentiable Simulation-Based System Identification](https://mlsanity.com/paper/2508.04696/) (2025)

## Abstract (arXiv)

Deep reinforcement learning (DRL) is a promising approach for developing legged locomotion skills. However, the iterative design process that is inevitable in practice is poorly supported by the default methodology. It is difficult to predict the outcomes of changes made to the reward functions, policy architectures, and the set of tasks being trained on. In this paper, we propose a practical method that allows the reward function to be fully redefined on each successive design iteration while limiting the deviation from the previous iteration. We characterize policies via sets of Deterministic Action Stochastic State (DASS) tuples, which represent the deterministic policy state-action pairs as sampled from the states visited by the trained stochastic policy. New policies are trained using a policy gradient algorithm which then mixes RL-based policy gradients with gradient updates defined by the DASS tuples. The tuples also allow for robust policy distillation to new network architectures. We demonstrate the effectiveness of this iterative-design approach on the bipedal robot Cassie, achieving stable walking with different gait styles at various speeds. We demonstrate the successful transfer of policies learned in simulation to the physical robot without any dynamics randomization, and that variable-speed walking policies for the physical robot can be represented by a small dataset of 5-10k tuples.