# Natural and Robust Walking using Reinforcement Learning without Demonstrations in High-Dimensional Musculoskeletal Models

arXiv: https://arxiv.org/abs/2309.02976 · HTML: https://mlsanity.com/paper/2309.02976/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 53 references that are in the map.
- First posted on arXiv 6 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Accelerated Policy Learning with Parallel Differentiable Simulation](https://mlsanity.com/paper/2204.07137/) (2022)
- [MyoSuite -- A contact-rich simulation suite for musculoskeletal motor control](https://mlsanity.com/paper/2205.13600/) (2022)
- [Advanced Skills by Learning Locomotion and Local Navigation End-to-End](https://mlsanity.com/paper/2209.12827/) (2022)
- [Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality](https://mlsanity.com/paper/2205.13521/) (2022)
- [DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems](https://mlsanity.com/paper/2206.00484/) (2022)
- [Latent Exploration for Reinforcement Learning](https://mlsanity.com/paper/2305.20065/) (2023)
- [Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments](https://mlsanity.com/paper/1804.00361/) (2018)
- [Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking](https://mlsanity.com/paper/2011.07537/) (2020)
- [SAR: Generalization of Physiological Agility and Dexterity via Synergistic Action Representation](https://mlsanity.com/paper/2307.03716/) (2023)
- [Generative GaitNet](https://mlsanity.com/paper/2201.12044/) (2022)
- [Learning with Muscles: Benefits for Data-Efficiency and Robustness in Anthropomorphic Tasks](https://mlsanity.com/paper/2207.03952/) (2022)

## Built on by (most cited first, 5 of 5)

- [Learning Speed-Adaptive Walking Agent Using Imitation Learning with Physics-Informed Simulation](https://mlsanity.com/paper/2412.03949/) (2024)
- [Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow](https://mlsanity.com/paper/2601.19707/) (2026)
- [Generating Realistic Arm Movements in Reinforcement Learning: A Quantitative Comparison of Reward Terms and Task Requirements](https://mlsanity.com/paper/2402.13949/) (2024)
- [Exciting Action: Investigating Efficient Exploration for Learning Musculoskeletal Humanoid Locomotion](https://mlsanity.com/paper/2407.11658/) (2024)
- [FastDSAC: Unlocking the Potential of Maximum Entropy RL in High-Dimensional Humanoid Control](https://mlsanity.com/paper/2603.12612/) (2026)

## Abstract (arXiv)

Humans excel at robust bipedal walking in complex natural environments. In each step, they adequately tune the interaction of biomechanical muscle dynamics and neuronal signals to be robust against uncertainties in ground conditions. However, it is still not fully understood how the nervous system resolves the musculoskeletal redundancy to solve the multi-objective control problem considering stability, robustness, and energy efficiency. In computer simulations, energy minimization has been shown to be a successful optimization target, reproducing natural walking with trajectory optimization or reflex-based control methods. However, these methods focus on particular motions at a time and the resulting controllers are limited when compensating for perturbations. In robotics, reinforcement learning~(RL) methods recently achieved highly stable (and efficient) locomotion on quadruped systems, but the generation of human-like walking with bipedal biomechanical models has required extensive use of expert data sets. This strong reliance on demonstrations often results in brittle policies and limits the application to new behaviors, especially considering the potential variety of movements for high-dimensional musculoskeletal models in 3D. Achieving natural locomotion with RL without sacrificing its incredible robustness might pave the way for a novel approach to studying human walking in complex natural environments. Videos: https://sites.google.com/view/naturalwalkingrl