# Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?

arXiv: https://arxiv.org/abs/1611.01055 · HTML: https://mlsanity.com/paper/1611.01055/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Nov 2016.

## Built on by (most cited first, 30 of 48)

- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [SimPoE: Simulated Character Control for 3D Human Pose Estimation](https://mlsanity.com/paper/2104.00683/) (2021)
- [Fast and Efficient Locomotion via Learned Gait Transitions](https://mlsanity.com/paper/2104.04644/) (2021)
- [Factory: Fast Contact for Robotic Assembly](https://mlsanity.com/paper/2205.03532/) (2022)
- [Learning to Jump from Pixels](https://mlsanity.com/paper/2110.15344/) (2021)
- [Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models](https://mlsanity.com/paper/2308.15143/) (2023)
- [Learning Torque Control for Quadrupedal Locomotion](https://mlsanity.com/paper/2203.05194/) (2022)
- [On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer](https://mlsanity.com/paper/2312.03673/) (2023)
- [PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision](https://mlsanity.com/paper/2203.15625/) (2022)
- [Robust High-speed Running for Quadruped Robots via Deep Reinforcement Learning](https://mlsanity.com/paper/2103.06484/) (2021)
- [HUMOS: Human Motion Model Conditioned on Body Shape](https://mlsanity.com/paper/2409.03944/) (2024)
- [Model-free Reinforcement Learning for Robust Locomotion using Demonstrations from Trajectory Optimization](https://mlsanity.com/paper/2107.06629/) (2021)
- [Learning and Deploying Robust Locomotion Policies with Minimal Dynamics Randomization](https://mlsanity.com/paper/2209.12878/) (2022)
- [Controlling the Solo12 Quadruped Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2309.16683/) (2023)
- [A Survey on Reinforcement Learning Methods in Character Animation](https://mlsanity.com/paper/2203.04735/) (2022)
- [Learning Low-Frequency Motion Control for Robust and Dynamic Robot Locomotion](https://mlsanity.com/paper/2209.14887/) (2022)
- [Generative GaitNet](https://mlsanity.com/paper/2201.12044/) (2022)
- [Learning to Get Up](https://mlsanity.com/paper/2205.00307/) (2022)
- [Shape My Moves: Text-Driven Shape-Aware Synthesis of Human Motions](https://mlsanity.com/paper/2504.03639/) (2025)
- [OstrichRL: A Musculoskeletal Ostrich Simulation to Study Bio-mechanical Locomotion](https://mlsanity.com/paper/2112.06061/) (2021)
- [Learning Bipedal Robot Locomotion from Human Movement](https://mlsanity.com/paper/2105.12277/) (2021)
- [DecAP: Decaying Action Priors for Accelerated Imitation Learning of Torque-Based Legged Locomotion Policies](https://mlsanity.com/paper/2310.05714/) (2023)
- [Learning the Optimal Power Flow: Environment Design Matters](https://mlsanity.com/paper/2403.17831/) (2024)
- [Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey](https://mlsanity.com/paper/2404.17070/) (2024)
- [Residual-MPPI: Online Policy Customization for Continuous Control](https://mlsanity.com/paper/2407.00898/) (2024)
- [Automatic Environment Shaping is the Next Frontier in RL](https://mlsanity.com/paper/2407.16186/) (2024)
- [Learning with Muscles: Benefits for Data-Efficiency and Robustness in Anthropomorphic Tasks](https://mlsanity.com/paper/2207.03952/) (2022)
- [CLAS: Coordinating Multi-Robot Manipulation with Central Latent Action Spaces](https://mlsanity.com/paper/2211.15824/) (2022)
- [Identifying Important Sensory Feedback for Learning Locomotion Skills](https://mlsanity.com/paper/2306.17101/) (2023)
- [Learning Robotic Manipulation Skills Using an Adaptive Force-Impedance Action Space](https://mlsanity.com/paper/2110.09904/) (2021)

## Abstract (arXiv)

The use of deep reinforcement learning allows for high-dimensional state descriptors, but little is known about how the choice of action representation impacts the learning difficulty and the resulting performance. We compare the impact of four different action parameterizations (torques, muscle-activations, target joint angles, and target joint-angle velocities) in terms of learning time, policy robustness, motion quality, and policy query rates. Our results are evaluated on a gait-cycle imitation task for multiple planar articulated figures and multiple gaits. We demonstrate that the local feedback provided by higher-level action parameterizations can significantly impact the learning, robustness, and quality of the resulting policies.