# Decoding ECoG signal into 3D hand translation using deep learning

arXiv: https://arxiv.org/abs/2110.03528 · HTML: https://mlsanity.com/paper/2110.03528/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 103 references that are in the map.
- First posted on arXiv 5 Oct 2021.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [Deep learning-based electroencephalography analysis: a systematic review](https://mlsanity.com/paper/1901.05498/) (2019)
- [Online Deep Learning: Learning Deep Neural Networks on the Fly](https://mlsanity.com/paper/1711.03705/) (2017)
- [Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks](https://mlsanity.com/paper/1511.06448/) (2015)
- [Effective and Efficient Dropout for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1904.03392/) (2019)
- [Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks](https://mlsanity.com/paper/1905.05928/) (2019)

## Built on by (most cited first, 5 of 5)

- [Impact of dataset size and long-term ECoG-based BCI usage on deep learning decoders performance](https://mlsanity.com/paper/2209.03789/) (2022)
- [Deep learning for ECoG brain-computer interface: end-to-end vs. hand-crafted features](https://mlsanity.com/paper/2210.02544/) (2022)
- [FingerFlex: Inferring Finger Trajectories from ECoG signals](https://mlsanity.com/paper/2211.01960/) (2022)
- [Multiscale fusion enhanced spiking neural network for invasive BCI neural signal decoding](https://mlsanity.com/paper/2410.03533/) (2024)
- [BrainDistill: Implantable Motor Decoding with Task-Specific Knowledge Distillation](https://mlsanity.com/paper/2601.17625/) (2026)

## Abstract (arXiv)

Motor brain-computer interfaces (BCIs) are a promising technology that may enable motor-impaired people to interact with their environment. Designing real-time and accurate BCI is crucial to make such devices useful, safe, and easy to use by patients in a real-life environment. Electrocorticography (ECoG)-based BCIs emerge as a good compromise between invasiveness of the recording device and good spatial and temporal resolution of the recorded signal. However, most ECoG signal decoders used to predict continuous hand movements are linear models. These models have a limited representational capacity and may fail to capture the relationship between ECoG signal and continuous hand movements. Deep learning (DL) models, which are state-of-the-art in many problems, could be a solution to better capture this relationship. In this study, we tested several DL-based architectures to predict imagined 3D continuous hand translation using time-frequency features extracted from ECoG signals. The dataset used in the analysis is a part of a long-term clinical trial (ClinicalTrials.gov identifier: NCT02550522) and was acquired during a closed-loop experiment with a tetraplegic subject. The proposed architectures include multilayer perceptron (MLP), convolutional neural networks (CNN), and long short-term memory networks (LSTM). The accuracy of the DL-based and multilinear models was compared offline using cosine similarity. Our results show that CNN-based architectures outperform the current state-of-the-art multilinear model. The best architecture exploited the spatial correlation between neighboring electrodes with CNN and benefited from the sequential character of the desired hand trajectory by using LSTMs. Overall, DL increased the average cosine similarity, compared to the multilinear model, by up to 60%, from 0.189 to 0.302 and from 0.157 to 0.249 for the left and right hand, respectively.