# Transformer-based Spatial-Temporal Feature Learning for EEG Decoding

arXiv: https://arxiv.org/abs/2106.11170 · HTML: https://mlsanity.com/paper/2106.11170/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 54 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [TSception: Capturing Temporal Dynamics and Spatial Asymmetry from EEG for Emotion Recognition](https://mlsanity.com/paper/2104.02935/) (2021)
- [Spatial-Temporal Recurrent Neural Network for Emotion Recognition](https://mlsanity.com/paper/1705.04515/) (2017)

## Built on by (most cited first, 30 of 97)

- [Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI](https://mlsanity.com/paper/2405.18765/) (2024)
- [CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding](https://mlsanity.com/paper/2412.07236/) (2024)
- [NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals](https://mlsanity.com/paper/2409.00101/) (2024)
- [REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjects](https://mlsanity.com/paper/2510.21585/) (2025)
- [CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding](https://mlsanity.com/paper/2506.23075/) (2025)
- [EEG Foundation Models: A Critical Review of Current Progress and Future Directions](https://mlsanity.com/paper/2507.11783/) (2025)
- [CodeBrain: Bridging Decoupled Tokenizer and Multi-Scale Architecture for EEG Foundation Model](https://mlsanity.com/paper/2506.09110/) (2025)
- [BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signals](https://mlsanity.com/paper/2505.18185/) (2025)
- [LUNA: Efficient and Topology-Agnostic Foundation Model for EEG Signal Analysis](https://mlsanity.com/paper/2510.22257/) (2025)
- [Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?](https://mlsanity.com/paper/2301.09017/) (2023)
- [BIOT: Cross-data Biosignal Learning in the Wild](https://mlsanity.com/paper/2305.10351/) (2023)
- [Tokenizing Single-Channel EEG with Time-Frequency Motif Learning](https://mlsanity.com/paper/2502.16060/) (2025)
- [AdaBrain-Bench: Benchmarking Brain Foundation Models for Brain-Computer Interface Applications](https://mlsanity.com/paper/2507.09882/) (2025)
- [CEReBrO: Compact Encoder for Representations of Brain Oscillations Using Efficient Alternating Attention](https://mlsanity.com/paper/2501.10885/) (2025)
- [CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models](https://mlsanity.com/paper/2503.07667/) (2025)
- [NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping](https://mlsanity.com/paper/2410.05341/) (2024)
- [NeurIPT: Foundation Model for Neural Interfaces](https://mlsanity.com/paper/2510.16548/) (2025)
- [TraHGR: Transformer for Hand Gesture Recognition via ElectroMyography](https://mlsanity.com/paper/2203.16336/) (2022)
- [Uni-NTFM: A Unified Foundation Model for EEG Signal Representation Learning](https://mlsanity.com/paper/2509.24222/) (2025)
- [DeeperBrain: A Neuro-Grounded EEG Foundation Model Towards Universal BCI](https://mlsanity.com/paper/2601.06134/) (2026)
- [GeoECG: Data Augmentation via Wasserstein Geodesic Perturbation for Robust Electrocardiogram Prediction](https://mlsanity.com/paper/2208.01220/) (2022)
- [TEMGNet: Deep Transformer-based Decoding of Upperlimb sEMG for Hand Gestures Recognition](https://mlsanity.com/paper/2109.12379/) (2021)
- [Trends in Machine Learning and Electroencephalogram (EEG): A Review for Undergraduate Researchers](https://mlsanity.com/paper/2307.02819/) (2023)
- [A Strong and Simple Deep Learning Baseline for BCI MI Decoding](https://mlsanity.com/paper/2309.07159/) (2023)
- [AnyECG: Foundational Models for Multitask Cardiac Analysis in Real-World Settings](https://mlsanity.com/paper/2411.17711/) (2024)
- [BrainRVQ: A High-Fidelity EEG Foundation Model via Dual-Domain Residual Quantization and Hierarchical Autoregression](https://mlsanity.com/paper/2602.16951/) (2026)
- [A Knowledge-Driven Cross-view Contrastive Learning for EEG Representation](https://mlsanity.com/paper/2310.03747/) (2023)
- [BioSerenity-E1: a self-supervised EEG model for medical applications](https://mlsanity.com/paper/2503.10362/) (2025)
- [Pretraining Large Brain Language Model for Active BCI: Silent Speech](https://mlsanity.com/paper/2504.21214/) (2025)
- [EEGDM: EEG Representation Learning via Generative Diffusion Model](https://mlsanity.com/paper/2508.14086/) (2025)

## Abstract (arXiv)

At present, people usually use some methods based on convolutional neural networks (CNNs) for Electroencephalograph (EEG) decoding. However, CNNs have limitations in perceiving global dependencies, which is not adequate for common EEG paradigms with a strong overall relationship. Regarding this issue, we propose a novel EEG decoding method that mainly relies on the attention mechanism. The EEG data is firstly preprocessed and spatially filtered. And then, we apply attention transforming on the feature-channel dimension so that the model can enhance more relevant spatial features. The most crucial step is to slice the data in the time dimension for attention transforming, and finally obtain a highly distinguishable representation. At this time, global averaging pooling and a simple fully-connected layer are used to classify different categories of EEG data. Experiments on two public datasets indicate that the strategy of attention transforming effectively utilizes spatial and temporal features. And we have reached the level of the state-of-the-art in multi-classification of EEG, with fewer parameters. As far as we know, it is the first time that a detailed and complete method based on the transformer idea has been proposed in this field. It has good potential to promote the practicality of brain-computer interface (BCI). The source code can be found at: textithttps://github.com/anranknight/EEG-Transformer.