# Copilot-Assisted Second-Thought Framework for Brain-to-Robot Hand Motion Decoding

arXiv: https://arxiv.org/abs/2603.27492 · HTML: https://mlsanity.com/paper/2603.27492/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 41 references that are in the map.
- First posted on arXiv 29 Mar 2026.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [Transformer-based Spatial-Temporal Feature Learning for EEG Decoding](https://mlsanity.com/paper/2106.11170/) (2021)
- [EEG-TCNet: An Accurate Temporal Convolutional Network for Embedded Motor-Imagery Brain-Machine Interfaces](https://mlsanity.com/paper/2006.00622/) (2020)
- [Towards the Next Generation of Retinal Neuroprosthesis: Visual Computation with Spikes](https://mlsanity.com/paper/2001.04064/) (2020)
- [Comparison of Attention-based Deep Learning Models for EEG Classification](https://mlsanity.com/paper/2012.01074/) (2020)
- [BiCurNet: Pre-Movement EEG based Neural Decoder for Biceps Curl Trajectory Estimation](https://mlsanity.com/paper/2301.03965/) (2023)
- [EEG-ConvTransformer for Single-Trial EEG based Visual Stimuli Classification](https://mlsanity.com/paper/2107.03983/) (2021)
- [EEG Cortical Source Feature based Hand Kinematics Decoding using Residual CNN-LSTM Neural Network](https://mlsanity.com/paper/2304.06321/) (2023)

## Abstract (arXiv)

Motor kinematics prediction (MKP) from electroencephalography (EEG) is an important research area for developing movement-related brain-computer interfaces (BCIs). While traditional methods often rely on convolutional neural networks (CNNs) or recurrent neural networks (RNNs), Transformer-based models have shown strong ability in modeling long sequential EEG data. In this study, we propose a CNN-attention hybrid model for decoding hand kinematics from EEG during grasp-and-lift tasks, achieving strong performance in within-subject experiments. We further extend this approach to EEG-EMG multimodal decoding, which yields substantially improved results. Within-subject tests achieve PCC values of 0.9854, 0.9946, and 0.9065 for the X, Y, and Z axes, respectively, computed on the midpoint trajectory between the thumb and index finger, while cross-subject tests result in 0.9643, 0.9795, and 0.5852. The decoded trajectories from both modalities are then used to control a Franka Panda robotic arm in a MuJoCo simulation. To enhance trajectory fidelity, we introduce a copilot framework that filters low-confidence decoded points using a motion-state-aware critic within a finite-state machine. This post-processing step improves the overall within-subject PCC of EEG-only decoding to 0.93 while excluding fewer than 20% of the data points.