# Decoding EEG Speech Perception with Transformers and VAE-based Data Augmentation

arXiv: https://arxiv.org/abs/2501.04359 · HTML: https://mlsanity.com/paper/2501.04359/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 21 references that are in the map.
- First posted on arXiv 8 Jan 2025.

## Builds on (most cited first, 5 of 5)

- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Digital Voicing of Silent Speech](https://mlsanity.com/paper/2010.02960/) (2020)
- [A Cross-Modal Approach to Silent Speech with LLM-Enhanced Recognition](https://mlsanity.com/paper/2403.05583/) (2024)
- [Classification of Imagined Speech Using Siamese Neural Network](https://mlsanity.com/paper/2008.12487/) (2020)

## Built on by (most cited first, 4 of 4)

- [A Survey on Bridging EEG Signals and Generative AI: From Image and Text to Beyond](https://mlsanity.com/paper/2502.12048/) (2025)
- [Neural Brain Fields: A NeRF-Inspired Approach for Generating Nonexistent EEG Electrodes](https://mlsanity.com/paper/2601.00012/) (2025)
- [MultiDiffNet: A Multi-Objective Diffusion Framework for Generalizable Brain Decoding](https://mlsanity.com/paper/2511.18294/) (2025)
- [Synthetic Data Generation for Brain-Computer Interfaces: Overview, Benchmarking, and Future Directions](https://mlsanity.com/paper/2603.12296/) (2026)

## Abstract (arXiv)

Decoding speech from non-invasive brain signals, such as electroencephalography (EEG), has the potential to advance brain-computer interfaces (BCIs), with applications in silent communication and assistive technologies for individuals with speech impairments. However, EEG-based speech decoding faces major challenges, such as noisy data, limited datasets, and poor performance on complex tasks like speech perception. This study attempts to address these challenges by employing variational autoencoders (VAEs) for EEG data augmentation to improve data quality and applying a state-of-the-art (SOTA) sequence-to-sequence deep learning architecture, originally successful in electromyography (EMG) tasks, to EEG-based speech decoding. Additionally, we adapt this architecture for word classification tasks. Using the Brennan dataset, which contains EEG recordings of subjects listening to narrated speech, we preprocess the data and evaluate both classification and sequence-to-sequence models for EEG-to-words/sentences tasks. Our experiments show that VAEs have the potential to reconstruct artificial EEG data for augmentation. Meanwhile, our sequence-to-sequence model achieves more promising performance in generating sentences compared to our classification model, though both remain challenging tasks. These findings lay the groundwork for future research on EEG speech perception decoding, with possible extensions to speech production tasks such as silent or imagined speech.