# Cross-Subject Generalization in Decoding Perceived Speech from Non-Invasive Brain Recordings

arXiv: https://arxiv.org/abs/2608.22420 · HTML: https://mlsanity.com/paper/2608.22420/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 70 references that are in the map.
- First posted on arXiv 23 Aug 2026.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI](https://mlsanity.com/paper/2405.18765/) (2024)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding](https://mlsanity.com/paper/2412.07236/) (2024)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling](https://mlsanity.com/paper/2302.00861/) (2023)
- [Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion](https://mlsanity.com/paper/2403.07721/) (2024)
- [Brain decoding: toward real-time reconstruction of visual perception](https://mlsanity.com/paper/2310.19812/) (2023)
- [BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signals](https://mlsanity.com/paper/2505.18185/) (2025)
- [Domain-Invariant Representation Learning from EEG with Private Encoders](https://mlsanity.com/paper/2201.11613/) (2022)
- [ConvConcatNet: a deep convolutional neural network to reconstruct mel spectrogram from the EEG](https://mlsanity.com/paper/2401.04965/) (2024)
- [Open-vocabulary Auditory Neural Decoding Using fMRI-prompted LLM](https://mlsanity.com/paper/2405.07840/) (2024)
- [Beware of Overestimated Decoding Performance Arising from Temporal Autocorrelations in Electroencephalogram Signals](https://mlsanity.com/paper/2405.17024/) (2024)
- [Correlated Components Analysis - Extracting Reliable Dimensions in Multivariate Data](https://mlsanity.com/paper/1801.08881/) (2018)
- [Semantic reconstruction of continuous language from MEG signals](https://mlsanity.com/paper/2309.07701/) (2023)
- [Contrastive Learning of Shared Spatiotemporal EEG Representations Across Individuals for Naturalistic Neuroscience](https://mlsanity.com/paper/2402.14213/) (2024)

## Built on by (most cited first, 2 of 2)

- [Brain-to-Language Decoding: Tasks, Signals, Methods, Evaluation, Practical Use and Beyond](https://mlsanity.com/paper/2609.27650/) (2026)
- [Subject-Invariant Cross-Modal Decoding of Perceived Speech from Brain Recordings](https://mlsanity.com/paper/2609.30832/) (2026)

## Abstract (arXiv)

Decoding perceived speech from non-invasive brain recordings has garnered significant attention in recent years due to its wide range of potential applications. However, existing methods face considerable challenges in cross-subject decoding, primarily due to limited generalizability and the absence of explicit mechanisms for extracting subject-consistent information. These limitations result in high training costs and suboptimal decoding performance. To address these challenges, we propose an innovative Cross-Subject Perceived Speech Decoding (CPSD) framework, which comprises two training stages: source model pre-training and personal specialization. In the source model pre-training stage, contrastive learning is employed to capture shared representations across multiple source subjects. Subsequently, personal specialization initializes the model for the target subject by extracting consistent components from the source model and fine-tuning it using target subject data. Additionally, we introduce the Positional Encoding-based Spatial Attention (PESA) module, which remaps MEG/EEG data into a standardized reference space, thereby enhancing cross-subject consistency and facilitating model training. We evaluate the proposed CPSD framework on three perceived speech neural datasets encompassing different modalities and languages. The results demonstrate that our framework outperforms baseline methods by more than 6.8%, 15.4%, and 15.8% in Top-10 accuracy on the Armeni 2022, PKUEEG 2025, and Broderick 2018 datasets, respectively. Further analyses confirm the effectiveness, efficiency, and robustness of the proposed approach.