mlsanityData to 7 Oct 2026

DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems

arXiv 2206.00484 ↗First posted 30 May 2022cs.RO cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

Muscle-actuated organisms are capable of learning an unparalleled diversity of dexterous movements despite their vast amount of muscles. Reinforcement learning (RL) on large musculoskeletal models, however, has not been able to show similar performance. We conjecture that ineffective exploration in large overactuated action spaces is a key problem. This is supported by the finding that common exploration noise strategies are inadequate in synthetic examples of overactuated systems. We identify differential extrinsic plasticity (DEP), a method from the domain of self-organization, as being able to induce state-space covering exploration within seconds of interaction. By integrating DEP into RL, we achieve fast learning of reaching and locomotion in musculoskeletal systems, outperforming current approaches in all considered tasks in sample efficiency and robustness.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 21 of its 68 references that are in the map. First posted on arXiv 30 May 2022.

Builds on

21 of 68 references.

References · 21

  1. 2017Hindsight Experience Replaycited by 1,096
  2. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  3. 2018Maximum a Posteriori Policy Optimisationcited by 318
  4. 2020First return, then explorecited by 256
  5. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 158
  6. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  7. 2020Sample-efficient Cross-Entropy Method for Real-time Planningcited by 90
  8. 2019Growing Action Spacescited by 29
  9. 2019Synthesis of Biologically Realistic Human Motion Using Joint Torque Actuationcited by 23
  10. 2020Learning to Represent Action Values as a Hypergraph on the Action Verticescited by 22
  11. 2021When should agents explore?cited by 20
  12. 2018Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environmentscited by 19
9 more
  1. 2020Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarkingcited by 18
  2. 2019Artificial Intelligence for Prosthetics - challenge solutionscited by 12
  3. 2020Reinforcement Learning Control of a Biomechanical Model of the Upper Extremitycited by 10
  4. 2021Action Priors for Large Action Spaces in Roboticscited by 10
  5. 2021OstrichRL: A Musculoskeletal Ostrich Simulation to Study Bio-mechanical Locomotioncited by 10
  6. 2020Reinforcement Learning of Musculoskeletal Control from Functional Simulationscited by 8
  7. 2013Information driven self-organization of complex robotic behaviorscited by 4
  8. 2015A novel plasticity rule can explain the development of sensorimotor intelligencecited by 3
  9. 2020How do Offline Measures for Exploration in Reinforcement Learning behave?cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 5; 2025: 9; 2026: 7.

27 papers.

Sort:

By area:

  1. 2023Latent Exploration for Reinforcement Learningcited by 21
  2. 2024EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Visioncited by 18
  3. 2023SAR: Generalization of Physiological Agility and Dexterity via Synergistic Action Representationcited by 17
  4. 2023Self Model for Embodied Intelligence: Modeling Full-Body Human Musculoskeletal System and Locomotion Control with Hierarchical Low-Dimensional Representationcited by 16
  5. 2024Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learningcited by 12
  6. 2023MyoDex: A Generalizable Prior for Dexterous Manipulationcited by 11
  7. 2025KINESIS: Motion Imitation for Human Musculoskeletal Locomotioncited by 7
  8. 2022Learning with Muscles: Benefits for Data-Efficiency and Robustness in Anthropomorphic Taskscited by 5
  9. 2024Identifying Policy Gradient Subspacescited by 5
  10. 2025Differentiable Simulation of Hard Contacts with Soft Gradients for Learning and Controlcited by 5
  11. 2023Natural and Robust Walking using Reinforcement Learning without Demonstrations in High-Dimensional Musculoskeletal Modelscited by 4
  12. 2026Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flowcited by 4
15 more
  1. 2025Arnold: A multi-task, multi-embodiment muscle transformer policycited by 2
  2. 2024Exciting Action: Investigating Efficient Exploration for Learning Musculoskeletal Humanoid Locomotioncited by 1
  3. 2025Bipedal Balance Control with Whole-body Musculoskeletal Standing and Falling Simulationscited by 1
  4. 2025SSPINNpose: A Self-Supervised PINN for Inertial Pose and Dynamics Estimationcited by 1
  5. 2025Grounding Intelligence in Movementcited by 1
  6. 2026Scaling Whole-Body Human Musculoskeletal Behavior Emulation for Specificity and Diversitycited by 1
  7. 2024Replication of Impedance Identification Experiments on a Reinforcement-Learning-Controlled Digital Twin of Human Elbowscited by 0
  8. 2025Where to Intervene: Action Selection in Deep Reinforcement Learningcited by 0
  9. 2025Embodied sensorimotor control: computational modeling of the neural control of movementcited by 0
  10. 2025Embodied Learning of Reward for Musculoskeletal Control with Vision Language Modelscited by 0
  11. 2026MyoInteract: A Framework for Fast Prototyping of Biomechanical HCI Tasks using Reinforcement Learningcited by 0
  12. 2026Imitation Learning from Human Motion Alone Does Not Guarantee Biomechanically Plausible Gait Kineticscited by 0
  13. 2026Lambda-Hold Control: Human-Like Movement Emerges from a Minimal Task Reward in Predictive Musculoskeletal Simulationcited by 0
  14. 2026Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPOcited by 0
  15. 2026Reflex-Informed Neuromuscular Reinforcement Learning for Muscle-Driven Locomotioncited by 0