# Synthesizing Speech from Intracranial Depth Electrodes using an Encoder-Decoder Framework

arXiv: https://arxiv.org/abs/2111.01457 · HTML: https://mlsanity.com/paper/2111.01457/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 55 references that are in the map.
- First posted on arXiv 2 Nov 2021.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2008.06239/) (2020)
- [Dropout Feature Ranking for Deep Learning Models](https://mlsanity.com/paper/1712.08645/) (2017)

## Built on by (most cited first, 4 of 4)

- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [MiSTR: Multi-Modal iEEG-to-Speech Synthesis with Transformer-Based Prosody Prediction and Neural Phase Reconstruction](https://mlsanity.com/paper/2508.03166/) (2025)
- [Neural Decoding of Overt Speech from ECoG Using Vision Transformers and Contrastive Representation Learning](https://mlsanity.com/paper/2512.04618/) (2025)
- [Brain-to-Speech: Prosody Feature Engineering and Transformer-Based Reconstruction](https://mlsanity.com/paper/2604.05751/) (2026)

## Abstract (arXiv)

Speech Neuroprostheses have the potential to enable communication for people with dysarthria or anarthria. Recent advances have demonstrated high-quality text decoding and speech synthesis from electrocorticographic grids placed on the cortical surface. Here, we investigate a less invasive measurement modality in three participants, namely stereotactic EEG (sEEG) that provides sparse sampling from multiple brain regions, including subcortical regions. To evaluate whether sEEG can also be used to synthesize audio from neural recordings, we employ a recurrent encoder-decoder model based on modern deep learning methods. We find that speech can indeed be reconstructed with correlations up to 0.8 from these minimally invasive recordings, despite limited amounts of training data. In particular, the architecture we employ naturally picks up on the temporal nature of the data and thereby outperforms an existing benchmark based on non-regressive convolutional neural networks.