# Toward Fully-End-to-End Listened Speech Decoding from EEG Signals

arXiv: https://arxiv.org/abs/2406.08644 · HTML: https://mlsanity.com/paper/2406.08644/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 15 of 15)

- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Towards Voice Reconstruction from EEG during Imagined Speech](https://mlsanity.com/paper/2301.07173/) (2023)
- [Relating EEG to continuous speech using deep neural networks: a review](https://mlsanity.com/paper/2302.01736/) (2023)
- [ConvConcatNet: a deep convolutional neural network to reconstruct mel spectrogram from the EEG](https://mlsanity.com/paper/2401.04965/) (2024)
- [Knowledge-guided EEG Representation Learning](https://mlsanity.com/paper/2403.03222/) (2024)
- [Neural2Speech: A Transfer Learning Framework for Neural-Driven Speech Reconstruction](https://mlsanity.com/paper/2310.04644/) (2023)

## Built on by (most cited first, 8 of 8)

- [Neural Codecs as Biosignal Tokenizers](https://mlsanity.com/paper/2510.09095/) (2025)
- [Enhancing Listened Speech Decoding from EEG via Parallel Phoneme Sequence Prediction](https://mlsanity.com/paper/2501.04844/) (2025)
- [Articulatory Feature Prediction from Surface EMG during Speech Production](https://mlsanity.com/paper/2505.13814/) (2025)
- [A Survey on Bridging EEG Signals and Generative AI: From Image and Text to Beyond](https://mlsanity.com/paper/2502.12048/) (2025)
- [EEG-ReMinD: Enhancing Neurodegenerative EEG Decoding through Self-Supervised State Reconstruction-Primed Riemannian Dynamics](https://mlsanity.com/paper/2501.08139/) (2025)
- [An Approach to Simultaneous Acquisition of Real-Time MRI Video, EEG, and Surface EMG for Articulatory, Brain, and Muscle Activity During Speech Production](https://mlsanity.com/paper/2603.04840/) (2026)
- [Neuro2Semantic: A Transfer Learning Framework for Semantic Reconstruction of Continuous Language from Human Intracranial EEG](https://mlsanity.com/paper/2506.00381/) (2025)
- [MindVoice: Reconstructing Intelligible Speech from Non-invasive Neural Signals with Pretrained Priors](https://mlsanity.com/paper/2605.31173/) (2026)

## Abstract (arXiv)

Speech decoding from EEG signals is a challenging task, where brain activity is modeled to estimate salient characteristics of acoustic stimuli. We propose FESDE, a novel framework for Fully-End-to-end Speech Decoding from EEG signals. Our approach aims to directly reconstruct listened speech waveforms given EEG signals, where no intermediate acoustic feature processing step is required. The proposed method consists of an EEG module and a speech module along with a connector. The EEG module learns to better represent EEG signals, while the speech module generates speech waveforms from model representations. The connector learns to bridge the distributions of the latent spaces of EEG and speech. The proposed framework is both simple and efficient, by allowing single-step inference, and outperforms prior works on objective metrics. A fine-grained phoneme analysis is conducted to unveil model characteristics of speech decoding. The source code is available here: github.com/lee-jhwn/fesde.