# Brain-to-Speech: Prosody Feature Engineering and Transformer-Based Reconstruction

arXiv: https://arxiv.org/abs/2604.05751 · HTML: https://mlsanity.com/paper/2604.05751/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Builds on (most cited first, 11 of 11)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [Learning latent representations for style control and transfer in end-to-end speech synthesis](https://mlsanity.com/paper/1812.04342/) (2018)
- [Deep Learning-based Non-Intrusive Multi-Objective Speech Assessment Model with Cross-Domain Features](https://mlsanity.com/paper/2111.02363/) (2021)
- [Towards Voice Reconstruction from EEG during Imagined Speech](https://mlsanity.com/paper/2301.07173/) (2023)
- [Fine-grained style control in Transformer-based Text-to-speech Synthesis](https://mlsanity.com/paper/2110.06306/) (2021)
- [BigVSAN: Enhancing GAN-based Neural Vocoders with Slicing Adversarial Network](https://mlsanity.com/paper/2309.02836/) (2023)
- [GLA-Grad: A Griffin-Lim Extended Waveform Generation Diffusion Model](https://mlsanity.com/paper/2402.15516/) (2024)
- [Autovocoder: Fast Waveform Generation from a Learned Speech Representation using Differentiable Digital Signal Processing](https://mlsanity.com/paper/2211.06989/) (2022)
- [Synthesizing Speech from Intracranial Depth Electrodes using an Encoder-Decoder Framework](https://mlsanity.com/paper/2111.01457/) (2021)

## Abstract (arXiv)

This chapter presents a novel approach to brain-to-speech (BTS) synthesis from intracranial electroencephalography (iEEG) data, emphasizing prosody-aware feature engineering and advanced transformer-based models for high-fidelity speech reconstruction. Driven by the increasing interest in decoding speech directly from brain activity, this work integrates neuroscience, artificial intelligence, and signal processing to generate accurate and natural speech. We introduce a novel pipeline for extracting key prosodic features directly from complex brain iEEG signals, including intonation, pitch, and rhythm. To effectively utilize these crucial features for natural-sounding speech, we employ advanced deep learning models. Furthermore, this chapter introduces a novel transformer encoder architecture specifically designed for brain-to-speech tasks. Unlike conventional models, our architecture integrates the extracted prosodic features to significantly enhance speech reconstruction, resulting in generated speech with improved intelligibility and expressiveness. A detailed evaluation demonstrates superior performance over established baseline methods, such as traditional Griffin-Lim and CNN-based reconstruction, across both quantitative and perceptual metrics. By demonstrating these advancements in feature extraction and transformer-based learning, this chapter contributes to the growing field of AI-driven neuroprosthetics, paving the way for assistive technologies that restore communication for individuals with speech impairments. Finally, we discuss promising future research directions, including the integration of diffusion models and real-time inference systems.