# Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2010.04304 · HTML: https://mlsanity.com/paper/2010.04304/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2020.

## Built on by (most cited first, 27 of 27)

- [Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion](https://mlsanity.com/paper/2202.05481/) (2022)
- [Learning Torque Control for Quadrupedal Locomotion](https://mlsanity.com/paper/2203.05194/) (2022)
- [Critic Sequential Monte Carlo](https://mlsanity.com/paper/2205.15460/) (2022)
- [A Survey on Reinforcement Learning Methods in Character Animation](https://mlsanity.com/paper/2203.04735/) (2022)
- [Braxlines: Fast and Interactive Toolkit for RL-driven Behavior Engineering beyond Reward Maximization](https://mlsanity.com/paper/2110.04686/) (2021)
- [The Impact of Task Underspecification in Evaluating Deep Reinforcement Learning](https://mlsanity.com/paper/2210.08607/) (2022)
- [Physics-based Motion Retargeting from Sparse Inputs](https://mlsanity.com/paper/2307.01938/) (2023)
- [Learning the Optimal Power Flow: Environment Design Matters](https://mlsanity.com/paper/2403.17831/) (2024)
- [Learning with Muscles: Benefits for Data-Efficiency and Robustness in Anthropomorphic Tasks](https://mlsanity.com/paper/2207.03952/) (2022)
- [Identifying Important Sensory Feedback for Learning Locomotion Skills](https://mlsanity.com/paper/2306.17101/) (2023)
- [Towards Automatic Actor-Critic Solutions to Continuous Control](https://mlsanity.com/paper/2106.08918/) (2021)
- [Reinforcement Learning with Adaptive Curriculum Dynamics Randomization for Fault-Tolerant Robot Control](https://mlsanity.com/paper/2111.10005/) (2021)
- [Design of experiments for the calibration of history-dependent models via deep reinforcement learning and an enhanced Kalman filter](https://mlsanity.com/paper/2209.13126/) (2022)
- [Reinforcement Learning with Formal Performance Metrics for Quadcopter Attitude Control under Non-nominal Contexts](https://mlsanity.com/paper/2107.12942/) (2021)
- [TERC: A Transfer Entropy Redundancy Criterion for State Variable Selection in Reinforcement Learning](https://mlsanity.com/paper/2401.11512/) (2024)
- [Attention-based Reinforcement Learning for Combinatorial Optimization: Application to Job Shop Scheduling Problem](https://mlsanity.com/paper/2401.16580/) (2024)
- [How to Choose a Reinforcement-Learning Algorithm](https://mlsanity.com/paper/2407.20917/) (2024)
- [A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.17697/) (2025)
- [Understanding reinforcement learned crowds](https://mlsanity.com/paper/2209.09344/) (2022)
- [Investigating the Impact of Action Representations in Policy Gradient Algorithms](https://mlsanity.com/paper/2309.06921/) (2023)
- [Detecting danger in gridworlds using Gromov's Link Condition](https://mlsanity.com/paper/2201.06274/) (2022)
- [Attaining Human`s Desirable Outcomes in Human-AI Interaction via Structural Causal Games](https://mlsanity.com/paper/2405.16588/) (2024)
- [Multi-Step First: A Lightweight Deep Reinforcement Learning Strategy for Robust Continuous Control with Partial Observability](https://mlsanity.com/paper/2209.04999/) (2022)
- [Progress and summary of reinforcement learning on energy management of MPS-EV](https://mlsanity.com/paper/2211.04001/) (2022)
- [SDGym: Low-Code Reinforcement Learning Environments using System Dynamics Models](https://mlsanity.com/paper/2310.12494/) (2023)
- [A General Approach of Automated Environment Design for Learning the Optimal Power Flow](https://mlsanity.com/paper/2505.07832/) (2025)
- [A Benchmark Study of Deep Reinforcement Learning Algorithms for the Container Stowage Planning Problem](https://mlsanity.com/paper/2510.02589/) (2025)

## Abstract (arXiv)

Learning to locomote is one of the most common tasks in physics-based animation and deep reinforcement learning (RL). A learned policy is the product of the problem to be solved, as embodied by the RL environment, and the RL algorithm. While enormous attention has been devoted to RL algorithms, much less is known about the impact of design choices for the RL environment. In this paper, we show that environment design matters in significant ways and document how it can contribute to the brittle nature of many RL results. Specifically, we examine choices related to state representations, initial state distributions, reward structure, control frequency, episode termination procedures, curriculum usage, the action space, and the torque limits. We aim to stimulate discussion around such choices, which in practice strongly impact the success of RL when applied to continuous-action control problems of interest to animation, such as learning to locomote.