# Neuromuscular Reinforcement Learning to Actuate Human Limbs through FES

arXiv: https://arxiv.org/abs/2209.07849 · HTML: https://mlsanity.com/paper/2209.07849/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 10 references that are in the map.
- First posted on arXiv 16 Sep 2022.

## Builds on (most cited first, 5 of 5)

- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Neuromechanics-based Deep Reinforcement Learning of Neurostimulation Control in FES cycling](https://mlsanity.com/paper/2103.03057/) (2021)

## Built on by (most cited first, 4 of 4)

- [Towards AI-controlled FES-restoration of arm movements: neuromechanics-based reinforcement learning for 3-D reaching](https://mlsanity.com/paper/2301.04004/) (2023)
- [Towards AI-controlled FES-restoration of arm movements: Controlling for progressive muscular fatigue with Gaussian state-space models](https://mlsanity.com/paper/2301.04005/) (2023)
- [Towards AI-controlled FES-restoration of movements: Learning cycling stimulation pattern with reinforcement learning](https://mlsanity.com/paper/2303.09986/) (2023)
- [Physics-informed reinforcement learning via probabilistic co-adjustment functions](https://mlsanity.com/paper/2309.05404/) (2023)

## Abstract (arXiv)

Functional Electrical Stimulation (FES) is a technique to evoke muscle contraction through low-energy electrical signals. FES can animate paralysed limbs. Yet, an open challenge remains on how to apply FES to achieve desired movements. This challenge is accentuated by the complexities of human bodies and the non-stationarities of the muscles' responses. The former causes difficulties in performing inverse dynamics, and the latter causes control performance to degrade over extended periods of use. Here, we engage the challenge via a data-driven approach. Specifically, we learn to control FES through Reinforcement Learning (RL) which can automatically customise the stimulation for the patients. However, RL typically has Markovian assumptions while FES control systems are non-Markovian because of the non-stationarities. To deal with this problem, we use a recurrent neural network to create Markovian state representations. We cast FES controls into RL problems and train RL agents to control FES in different settings in both simulations and the real world. The results show that our RL controllers can maintain control performances over long periods and have better stimulation characteristics than PID controllers.