# Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments

arXiv: https://arxiv.org/abs/1804.00361 · HTML: https://mlsanity.com/paper/1804.00361/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2018.

## Built on by (most cited first, 19 of 19)

- [The Machine Learning for Combinatorial Optimization Competition (ML4CO): Results and Insights](https://mlsanity.com/paper/2203.02433/) (2022)
- [Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies](https://mlsanity.com/paper/2111.02552/) (2021)
- [Reward Gaming in Conditional Text Generation](https://mlsanity.com/paper/2211.08714/) (2022)
- [DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems](https://mlsanity.com/paper/2206.00484/) (2022)
- [A Survey on Reinforcement Learning Methods in Character Animation](https://mlsanity.com/paper/2203.04735/) (2022)
- [Generative GaitNet](https://mlsanity.com/paper/2201.12044/) (2022)
- [Perceptual Conversational Head Generation with Regularized Driver and Enhanced Renderer](https://mlsanity.com/paper/2206.12837/) (2022)
- [ML4CO-KIDA: Knowledge Inheritance in Dataset Aggregation](https://mlsanity.com/paper/2201.10328/) (2022)
- [Learning with Muscles: Benefits for Data-Efficiency and Robustness in Anthropomorphic Tasks](https://mlsanity.com/paper/2207.03952/) (2022)
- [Natural and Robust Walking using Reinforcement Learning without Demonstrations in High-Dimensional Musculoskeletal Models](https://mlsanity.com/paper/2309.02976/) (2023)
- [Bidirectional GaitNet: A Bidirectional Prediction Model of Human Gait and Anatomical Conditions](https://mlsanity.com/paper/2306.04161/) (2023)
- [Arnold: A multi-task, multi-embodiment muscle transformer policy](https://mlsanity.com/paper/2508.18066/) (2025)
- [Imitate then Transcend: Multi-Agent Optimal Execution with Dual-Window Denoise PPO](https://mlsanity.com/paper/2206.10736/) (2022)
- [Exciting Action: Investigating Efficient Exploration for Learning Musculoskeletal Humanoid Locomotion](https://mlsanity.com/paper/2407.11658/) (2024)
- [Incorporating Rivalry in Reinforcement Learning for a Competitive Game](https://mlsanity.com/paper/2208.10327/) (2022)
- [DCT: Dual Channel Training of Action Embeddings for Reinforcement Learning with Large Discrete Action Spaces](https://mlsanity.com/paper/2306.15913/) (2023)
- [Academic competitions](https://mlsanity.com/paper/2312.00268/) (2023)
- [Time Reversal Symmetry for Efficient Robotic Manipulations in Deep Reinforcement Learning](https://mlsanity.com/paper/2505.13925/) (2025)
- [Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO](https://mlsanity.com/paper/2608.24488/) (2026)

## Abstract (arXiv)

In the NIPS 2017 Learning to Run challenge, participants were tasked with building a controller for a musculoskeletal model to make it run as fast as possible through an obstacle course. Top participants were invited to describe their algorithms. In this work, we present eight solutions that used deep reinforcement learning approaches, based on algorithms such as Deep Deterministic Policy Gradient, Proximal Policy Optimization, and Trust Region Policy Optimization. Many solutions use similar relaxations and heuristics, such as reward shaping, frame skipping, discretization of the action space, symmetry, and policy blending. However, each of the eight teams implemented different modifications of the known algorithms.