# Neural2Speech: A Transfer Learning Framework for Neural-Driven Speech Reconstruction

arXiv: https://arxiv.org/abs/2310.04644 · HTML: https://mlsanity.com/paper/2310.04644/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2023.

## Built on by (most cited first, 4 of 4)

- [Toward Fully-End-to-End Listened Speech Decoding from EEG Signals](https://mlsanity.com/paper/2406.08644/) (2024)
- [Articulatory Feature Prediction from Surface EMG during Speech Production](https://mlsanity.com/paper/2505.13814/) (2025)
- [Neuro2Semantic: A Transfer Learning Framework for Semantic Reconstruction of Continuous Language from Human Intracranial EEG](https://mlsanity.com/paper/2506.00381/) (2025)
- [Intelligent Incident Hypertension Prediction in Obstructive Sleep Apnea](https://mlsanity.com/paper/2505.20615/) (2025)

## Abstract (arXiv)

Reconstructing natural speech from neural activity is vital for enabling direct communication via brain-computer interfaces. Previous efforts have explored the conversion of neural recordings into speech using complex deep neural network (DNN) models trained on extensive neural recording data, which is resource-intensive under regular clinical constraints. However, achieving satisfactory performance in reconstructing speech from limited-scale neural recordings has been challenging, mainly due to the complexity of speech representations and the neural data constraints. To overcome these challenges, we propose a novel transfer learning framework for neural-driven speech reconstruction, called Neural2Speech, which consists of two distinct training phases. First, a speech autoencoder is pre-trained on readily available speech corpora to decode speech waveforms from the encoded speech representations. Second, a lightweight adaptor is trained on the small-scale neural recordings to align the neural activity and the speech representation for decoding. Remarkably, our proposed Neural2Speech demonstrates the feasibility of neural-driven speech reconstruction even with only 20 minutes of intracranial data, which significantly outperforms existing baseline methods in terms of speech fidelity and intelligibility.