# NeuroSpex: Neuro-Guided Speaker Extraction with Cross-Modal Attention

arXiv: https://arxiv.org/abs/2409.02489 · HTML: https://mlsanity.com/paper/2409.02489/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 4 Sep 2024.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation](https://mlsanity.com/paper/1804.03619/) (2018)
- [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017)
- [SpEx: Multi-Scale Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2004.08326/) (2020)
- [EEG-based Auditory Attention Decoding: Towards Neuro-Steered Hearing Devices](https://mlsanity.com/paper/2008.04569/) (2020)
- [NeuroHeed: Neuro-Steered Speaker Extraction using EEG Signals](https://mlsanity.com/paper/2307.14303/) (2023)
- [NeuroHeed+: Improving Neuro-steered Speaker Extraction with Joint Auditory Attention Detection](https://mlsanity.com/paper/2312.07513/) (2023)
- [Speaker Extraction with Co-Speech Gestures Cue](https://mlsanity.com/paper/2203.16840/) (2022)
- [Spatially Selective Deep Non-linear Filters for Speaker Extraction](https://mlsanity.com/paper/2211.02420/) (2022)
- [BASEN: Time-Domain Brain-Assisted Speech Enhancement Network with Convolutional Cross Attention in Multi-talker Conditions](https://mlsanity.com/paper/2305.09994/) (2023)
- [Sparsity-Driven EEG Channel Selection for Brain-Assisted Speech Enhancement](https://mlsanity.com/paper/2311.13436/) (2023)

## Abstract (arXiv)

In the study of auditory attention, it has been revealed that there exists a robust correlation between attended speech and elicited neural responses, measurable through electroencephalography (EEG). Therefore, it is possible to use the attention information available within EEG signals to guide the extraction of the target speaker in a cocktail party computationally. In this paper, we present a neuro-guided speaker extraction model, i.e. NeuroSpex, using the EEG response of the listener as the sole auxiliary reference cue to extract attended speech from monaural speech mixtures. We propose a novel EEG signal encoder that captures the attention information. Additionally, we propose a cross-attention (CA) mechanism to enhance the speech feature representations, generating a speaker extraction mask. Experimental results on a publicly available dataset demonstrate that our proposed model outperforms two baseline models across various evaluation metrics.