# Spatio-Temporal Analysis of Transformer based Architecture for Attention Estimation from EEG

arXiv: https://arxiv.org/abs/2204.07162 · HTML: https://mlsanity.com/paper/2204.07162/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 42 references that are in the map.
- First posted on arXiv 4 Apr 2022.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [An overview of deep learning in medical imaging focusing on MRI](https://mlsanity.com/paper/1811.10052/) (2018)
- [EEG-Based Emotion Recognition Using Regularized Graph Neural Networks](https://mlsanity.com/paper/1907.07835/) (2019)
- [Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation](https://mlsanity.com/paper/2005.08575/) (2020)
- [Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks](https://mlsanity.com/paper/1511.06448/) (2015)
- [A Novel Bi-hemispheric Discrepancy Model for EEG Emotion Recognition](https://mlsanity.com/paper/1906.01704/) (2019)
- [A Multimodal Approach to Estimating Vigilance Using EEG and Forehead EOG](https://mlsanity.com/paper/1611.08492/) (2016)
- [Multi-channel EEG recordings during a sustained-attention driving task](https://mlsanity.com/paper/1809.06534/) (2018)

## Built on by (most cited first, 1 of 1)

- [Transformer-based EEG Decoding: A Survey](https://mlsanity.com/paper/2507.02320/) (2025)

## Abstract (arXiv)

For many years now, understanding the brain mechanism has been a great research subject in many different fields. Brain signal processing and especially electroencephalogram (EEG) has recently known a growing interest both in academia and industry. One of the main examples is the increasing number of Brain-Computer Interfaces (BCI) aiming to link brains and computers. In this paper, we present a novel framework allowing us to retrieve the attention state, i.e degree of attention given to a specific task, from EEG signals. While previous methods often consider the spatial relationship in EEG through electrodes and process them in recurrent or convolutional based architecture, we propose here to also exploit the spatial and temporal information with a transformer-based network that has already shown its supremacy in many machine-learning (ML) related studies, e.g. machine translation. In addition to this novel architecture, an extensive study on the feature extraction methods, frequential bands and temporal windows length has also been carried out. The proposed network has been trained and validated on two public datasets and achieves higher results compared to state-of-the-art models. As well as proposing better results, the framework could be used in real applications, e.g. Attention Deficit Hyperactivity Disorder (ADHD) symptoms or vigilance during a driving assessment.