# Exciting Action: Investigating Efficient Exploration for Learning Musculoskeletal Humanoid Locomotion

arXiv: https://arxiv.org/abs/2407.11658 · HTML: https://mlsanity.com/paper/2407.11658/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 20 of 20)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow](https://mlsanity.com/paper/1810.00821/) (2018)
- [MyoSuite -- A contact-rich simulation suite for musculoskeletal motor control](https://mlsanity.com/paper/2205.13600/) (2022)
- [What Matters for Adversarial Imitation Learning?](https://mlsanity.com/paper/2106.00672/) (2021)
- [Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?](https://mlsanity.com/paper/1611.01055/) (2016)
- [LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Locomotion](https://mlsanity.com/paper/2311.02496/) (2023)
- [DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems](https://mlsanity.com/paper/2206.00484/) (2022)
- [LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learning](https://mlsanity.com/paper/2303.00599/) (2023)
- [Latent Exploration for Reinforcement Learning](https://mlsanity.com/paper/2305.20065/) (2023)
- [Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments](https://mlsanity.com/paper/1804.00361/) (2018)
- [SAR: Generalization of Physiological Agility and Dexterity via Synergistic Action Representation](https://mlsanity.com/paper/2307.03716/) (2023)
- [Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning](https://mlsanity.com/paper/2103.17258/) (2021)
- [Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansion](https://mlsanity.com/paper/1912.05328/) (2019)
- [Artificial Intelligence for Prosthetics - challenge solutions](https://mlsanity.com/paper/1902.02441/) (2019)
- [Generative GaitNet](https://mlsanity.com/paper/2201.12044/) (2022)
- [Natural and Robust Walking using Reinforcement Learning without Demonstrations in High-Dimensional Musculoskeletal Models](https://mlsanity.com/paper/2309.02976/) (2023)

## Built on by (most cited first, 1 of 1)

- [Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow](https://mlsanity.com/paper/2601.19707/) (2026)

## Abstract (arXiv)

Learning a locomotion controller for a musculoskeletal system is challenging due to over-actuation and high-dimensional action space. While many reinforcement learning methods attempt to address this issue, they often struggle to learn human-like gaits because of the complexity involved in engineering an effective reward function. In this paper, we demonstrate that adversarial imitation learning can address this issue by analyzing key problems and providing solutions using both current literature and novel techniques. We validate our methodology by learning walking and running gaits on a simulated humanoid model with 16 degrees of freedom and 92 Muscle-Tendon Units, achieving natural-looking gaits with only a few demonstrations.