# DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems

arXiv: https://arxiv.org/abs/2206.00484 · HTML: https://mlsanity.com/paper/2206.00484/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 68 references that are in the map.
- First posted on arXiv 30 May 2022.

## Builds on (most cited first, 21 of 21)

- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Deep Reinforcement Learning in Large Discrete Action Spaces](https://mlsanity.com/paper/1512.07679/) (2015)
- [Parrot: Data-Driven Behavioral Priors for Reinforcement Learning](https://mlsanity.com/paper/2011.10024/) (2020)
- [Sample-efficient Cross-Entropy Method for Real-time Planning](https://mlsanity.com/paper/2008.06389/) (2020)
- [Growing Action Spaces](https://mlsanity.com/paper/1906.12266/) (2019)
- [Synthesis of Biologically Realistic Human Motion Using Joint Torque Actuation](https://mlsanity.com/paper/1904.13041/) (2019)
- [Learning to Represent Action Values as a Hypergraph on the Action Vertices](https://mlsanity.com/paper/2010.14680/) (2020)
- [When should agents explore?](https://mlsanity.com/paper/2108.11811/) (2021)
- [Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments](https://mlsanity.com/paper/1804.00361/) (2018)
- [Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking](https://mlsanity.com/paper/2011.07537/) (2020)
- [Artificial Intelligence for Prosthetics - challenge solutions](https://mlsanity.com/paper/1902.02441/) (2019)
- [Reinforcement Learning Control of a Biomechanical Model of the Upper Extremity](https://mlsanity.com/paper/2011.07105/) (2020)
- [Action Priors for Large Action Spaces in Robotics](https://mlsanity.com/paper/2101.04178/) (2021)
- [OstrichRL: A Musculoskeletal Ostrich Simulation to Study Bio-mechanical Locomotion](https://mlsanity.com/paper/2112.06061/) (2021)
- [Reinforcement Learning of Musculoskeletal Control from Functional Simulations](https://mlsanity.com/paper/2007.06669/) (2020)
- [Information driven self-organization of complex robotic behaviors](https://mlsanity.com/paper/1301.7473/) (2013)
- [A novel plasticity rule can explain the development of sensorimotor intelligence](https://mlsanity.com/paper/1505.00835/) (2015)
- [How do Offline Measures for Exploration in Reinforcement Learning behave?](https://mlsanity.com/paper/2010.15533/) (2020)

## Built on by (most cited first, 29 of 29)

- [Latent Exploration for Reinforcement Learning](https://mlsanity.com/paper/2305.20065/) (2023)
- [EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision](https://mlsanity.com/paper/2409.02224/) (2024)
- [SAR: Generalization of Physiological Agility and Dexterity via Synergistic Action Representation](https://mlsanity.com/paper/2307.03716/) (2023)
- [Self Model for Embodied Intelligence: Modeling Full-Body Human Musculoskeletal System and Locomotion Control with Hierarchical Low-Dimensional Representation](https://mlsanity.com/paper/2312.05473/) (2023)
- [Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning](https://mlsanity.com/paper/2401.11437/) (2024)
- [MyoDex: A Generalizable Prior for Dexterous Manipulation](https://mlsanity.com/paper/2309.03130/) (2023)
- [KINESIS: Motion Imitation for Human Musculoskeletal Locomotion](https://mlsanity.com/paper/2503.14637/) (2025)
- [Learning with Muscles: Benefits for Data-Efficiency and Robustness in Anthropomorphic Tasks](https://mlsanity.com/paper/2207.03952/) (2022)
- [Natural and Robust Walking using Reinforcement Learning without Demonstrations in High-Dimensional Musculoskeletal Models](https://mlsanity.com/paper/2309.02976/) (2023)
- [Identifying Policy Gradient Subspaces](https://mlsanity.com/paper/2401.06604/) (2024)
- [Differentiable Simulation of Hard Contacts with Soft Gradients for Learning and Control](https://mlsanity.com/paper/2506.14186/) (2025)
- [Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow](https://mlsanity.com/paper/2601.19707/) (2026)
- [Arnold: A multi-task, multi-embodiment muscle transformer policy](https://mlsanity.com/paper/2508.18066/) (2025)
- [Generating Realistic Arm Movements in Reinforcement Learning: A Quantitative Comparison of Reward Terms and Task Requirements](https://mlsanity.com/paper/2402.13949/) (2024)
- [Exciting Action: Investigating Efficient Exploration for Learning Musculoskeletal Humanoid Locomotion](https://mlsanity.com/paper/2407.11658/) (2024)
- [Bipedal Balance Control with Whole-body Musculoskeletal Standing and Falling Simulations](https://mlsanity.com/paper/2506.09383/) (2025)
- [SSPINNpose: A Self-Supervised PINN for Inertial Pose and Dynamics Estimation](https://mlsanity.com/paper/2506.11786/) (2025)
- [Grounding Intelligence in Movement](https://mlsanity.com/paper/2507.02771/) (2025)
- [Scaling Whole-Body Human Musculoskeletal Behavior Emulation for Specificity and Diversity](https://mlsanity.com/paper/2603.29332/) (2026)
- [Replication of Impedance Identification Experiments on a Reinforcement-Learning-Controlled Digital Twin of Human Elbows](https://mlsanity.com/paper/2402.02904/) (2024)
- [The Role of Tactile Sensing for Learning Reach and Grasp](https://mlsanity.com/paper/2502.20367/) (2025)
- [Where to Intervene: Action Selection in Deep Reinforcement Learning](https://mlsanity.com/paper/2507.04187/) (2025)
- [Embodied sensorimotor control: computational modeling of the neural control of movement](https://mlsanity.com/paper/2509.14360/) (2025)
- [Embodied Learning of Reward for Musculoskeletal Control with Vision Language Models](https://mlsanity.com/paper/2512.23077/) (2025)
- [MyoInteract: A Framework for Fast Prototyping of Biomechanical HCI Tasks using Reinforcement Learning](https://mlsanity.com/paper/2602.15245/) (2026)
- [Imitation Learning from Human Motion Alone Does Not Guarantee Biomechanically Plausible Gait Kinetics](https://mlsanity.com/paper/2603.12408/) (2026)
- [Lambda-Hold Control: Human-Like Movement Emerges from a Minimal Task Reward in Predictive Musculoskeletal Simulation](https://mlsanity.com/paper/2608.17030/) (2026)
- [Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO](https://mlsanity.com/paper/2608.24488/) (2026)
- [Reflex-Informed Neuromuscular Reinforcement Learning for Muscle-Driven Locomotion](https://mlsanity.com/paper/2609.11733/) (2026)

## Abstract (arXiv)

Muscle-actuated organisms are capable of learning an unparalleled diversity of dexterous movements despite their vast amount of muscles. Reinforcement learning (RL) on large musculoskeletal models, however, has not been able to show similar performance. We conjecture that ineffective exploration in large overactuated action spaces is a key problem. This is supported by the finding that common exploration noise strategies are inadequate in synthetic examples of overactuated systems. We identify differential extrinsic plasticity (DEP), a method from the domain of self-organization, as being able to induce state-space covering exploration within seconds of interaction. By integrating DEP into RL, we achieve fast learning of reaching and locomotion in musculoskeletal systems, outperforming current approaches in all considered tasks in sample efficiency and robustness.