# SSM2Mel: State Space Model to Reconstruct Mel Spectrogram from the EEG

arXiv: https://arxiv.org/abs/2501.10402 · HTML: https://mlsanity.com/paper/2501.10402/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jan 2025.

## Built on by (most cited first, 5 of 5)

- [Chirp Localization via Fine-Tuned Transformer Model: A Proof-of-Concept Study](https://mlsanity.com/paper/2503.22713/) (2025)
- [SHINE: Sequential Hierarchical Integration Network for EEG and MEG](https://mlsanity.com/paper/2602.23960/) (2026)
- [NEUROTOKEN: Joint Source and Directional AAD with Envelope Decoding via Conditional Flow Matching](https://mlsanity.com/paper/2610.00397/) (2026)
- [Decoding Stimulus Reconstruction-Based Auditory Attention Robustly in Unbalanced EEG Datasets](https://mlsanity.com/paper/2605.25605/) (2026)
- [Joint Text-Audio Alignment for EEG-to-Text Decoding in Chinese Speech Production and Perception](https://mlsanity.com/paper/2607.25626/) (2026)

## Abstract (arXiv)

Decoding speech from brain signals is a challenging research problem that holds significant importance for studying speech processing in the brain. Although breakthroughs have been made in reconstructing the mel spectrograms of audio stimuli perceived by subjects at the word or letter level using noninvasive electroencephalography (EEG), there is still a critical gap in precisely reconstructing continuous speech features, especially at the minute level. To address this issue, this paper proposes a State Space Model (SSM) to reconstruct the mel spectrogram of continuous speech from EEG, named SSM2Mel. This model introduces a novel Mamba module to effectively model the long sequence of EEG signals for imagined speech. In the SSM2Mel model, the S4-UNet structure is used to enhance the extraction of local features of EEG signals, and the Embedding Strength Modulator (ESM) module is used to incorporate subject-specific information. Experimental results show that our model achieves a Pearson correlation of 0.069 on the SparrKULee dataset, which is a 38% improvement over the previous baseline.