# Multi-class Decoding of Attended Speaker Direction Using Electroencephalogram and Audio Spatial Spectrum

arXiv: https://arxiv.org/abs/2411.06928 · HTML: https://mlsanity.com/paper/2411.06928/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 49 references that are in the map.
- First posted on arXiv 11 Nov 2024.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [ICLabel: An automated electroencephalographic independent component classifier, dataset, and website](https://mlsanity.com/paper/1901.07915/) (2019)
- [EEG-Deformer: A Dense Convolutional Transformer for Brain-computer Interfaces](https://mlsanity.com/paper/2405.00719/) (2024)
- [EEG-based Auditory Attention Decoding: Towards Neuro-Steered Hearing Devices](https://mlsanity.com/paper/2008.04569/) (2020)
- [Relating EEG to continuous speech using deep neural networks: a review](https://mlsanity.com/paper/2302.01736/) (2023)
- [Beware of Overestimated Decoding Performance Arising from Temporal Autocorrelations in Electroencephalogram Signals](https://mlsanity.com/paper/2405.17024/) (2024)
- [Extracting the Auditory Attention in a Dual-Speaker Scenario from EEG using a Joint CNN-LSTM Model](https://mlsanity.com/paper/2102.03957/) (2021)
- [Riemannian geometry-based decoding of the directional focus of auditory attention using EEG](https://mlsanity.com/paper/2010.07171/) (2020)
- [AADNet: An End-to-End Deep Learning Model for Auditory Attention Decoding](https://mlsanity.com/paper/2410.13059/) (2024)

## Built on by (most cited first, 1 of 1)

- [Decoding Stimulus Reconstruction-Based Auditory Attention Robustly in Unbalanced EEG Datasets](https://mlsanity.com/paper/2605.25605/) (2026)

## Abstract (arXiv)

Decoding the directional focus of an attended speaker from listeners' electroencephalogram (EEG) signals is essential for developing brain-computer interfaces to improve the quality of life for individuals with hearing impairment. Previous works have concentrated on binary directional focus decoding, i.e., determining whether the attended speaker is on the left or right side of the listener. However, a more precise decoding of the exact direction of the attended speaker is necessary for effective speech processing. Additionally, audio spatial information has not been effectively leveraged, resulting in suboptimal decoding results. In this paper, it is found that on the recently presented dataset with 14-class directional focus, models relying exclusively on EEG inputs exhibit significantly lower accuracy when decoding the directional focus in both leave-one-subject-out and leave-one-trial-out scenarios. By integrating audio spatial spectra with EEG features, the decoding accuracy can be effectively improved. The CNN, LSM-CNN, and Deformer models are employed to decode the directional focus from listeners' EEG signals and audio spatial spectra. The proposed Sp-EEG-Deformer model achieves notable 14-class decoding accuracies of 55.35% and 57.19% in leave-one-subject-out and leave-one-trial-out scenarios with a decision window of 1 second, respectively. Experiment results indicate increased decoding accuracy as the number of alternative directions reduces. These findings suggest the efficacy of our proposed dual modal directional focus decoding strategy.