# An LSTM Based Architecture to Relate Speech Stimulus to EEG

arXiv: https://arxiv.org/abs/2002.10988 · HTML: https://mlsanity.com/paper/2002.10988/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2020.

## Built on by (most cited first, 10 of 10)

- [End-to-end learnable EEG channel selection for deep neural networks with Gumbel-softmax](https://mlsanity.com/paper/2102.09050/) (2021)
- [DARNet: Dual Attention Refinement Network with Spatiotemporal Construction for Auditory Attention Detection](https://mlsanity.com/paper/2410.11181/) (2024)
- [Extracting the Auditory Attention in a Dual-Speaker Scenario from EEG using a Joint CNN-LSTM Model](https://mlsanity.com/paper/2102.03957/) (2021)
- [Learning Subject-Invariant Representations from Speech-Evoked EEG Using Variational Autoencoders](https://mlsanity.com/paper/2207.00323/) (2022)
- [Auditory Attention Decoding from EEG using Convolutional Recurrent Neural Network](https://mlsanity.com/paper/2103.02183/) (2021)
- [Predicting speech intelligibility from EEG in a non-linear classification paradigm](https://mlsanity.com/paper/2105.06844/) (2021)
- [Relate auditory speech to EEG by shallow-deep attention-based network](https://mlsanity.com/paper/2303.10897/) (2023)
- [AADNet: Exploring EEG Spatiotemporal Information for Fast and Accurate Orientation and Timbre Detection of Auditory Attention Based on A Cue-Masked Paradigm](https://mlsanity.com/paper/2501.03571/) (2025)
- [How Long short-term memory artificial neural network, synthetic data, and fine-tuning improve the classification of raw EEG data](https://mlsanity.com/paper/2604.04316/) (2026)
- [Merit: Making every modality earn its contribution in multimodal attention decoding](https://mlsanity.com/paper/2610.17507/) (2026)

## Abstract (arXiv)

Modeling the relationship between natural speech and a recorded electroencephalogram (EEG) helps us understand how the brain processes speech and has various applications in neuroscience and brain-computer interfaces. In this context, so far mainly linear models have been used. However, the decoding performance of the linear model is limited due to the complex and highly non-linear nature of the auditory processing in the human brain. We present a novel Long Short-Term Memory (LSTM)-based architecture as a non-linear model for the classification problem of whether a given pair of (EEG, speech envelope) correspond to each other or not. The model maps short segments of the EEG and the envelope to a common embedding space using a CNN in the EEG path and an LSTM in the speech path. The latter also compensates for the brain response delay. In addition, we use transfer learning to fine-tune the model for each subject. The mean classification accuracy of the proposed model reaches 85%, which is significantly higher than that of a state of the art Convolutional Neural Network (CNN)-based model (73%) and the linear model (69%).