# Enhancing Affective Representations of Music-Induced EEG through Multimodal Supervision and latent Domain Adaptation

arXiv: https://arxiv.org/abs/2202.09750 · HTML: https://mlsanity.com/paper/2202.09750/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 35 references that are in the map.
- First posted on arXiv 20 Feb 2022.

## Builds on (most cited first, 5 of 5)

- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [End-to-end learning for music audio tagging at scale](https://mlsanity.com/paper/1711.02520/) (2017)
- [Multimodal Metric Learning for Tag-based Music Retrieval](https://mlsanity.com/paper/2010.16030/) (2020)
- [Mind the beat: detecting audio onsets from EEG recordings of music listening](https://mlsanity.com/paper/2102.06393/) (2021)
- [Multiscale Fractal Analysis on EEG Signals for Music-Induced Emotion Recognition](https://mlsanity.com/paper/2010.16310/) (2020)

## Built on by (most cited first, 3 of 3)

- [Domain Adaptation and Generalization on Functional Medical Images: A Systematic Survey](https://mlsanity.com/paper/2212.03176/) (2022)
- [Predicting Artificial Neural Network Representations to Learn Recognition Model for Music Identification from Brain Recordings](https://mlsanity.com/paper/2412.15560/) (2024)
- [Bright Channel Prior Attention for Multispectral Pedestrian Detection](https://mlsanity.com/paper/2305.12845/) (2023)

## Abstract (arXiv)

The study of Music Cognition and neural responses to music has been invaluable in understanding human emotions. Brain signals, though, manifest a highly complex structure that makes processing and retrieving meaningful features challenging, particularly of abstract constructs like affect. Moreover, the performance of learning models is undermined by the limited amount of available neuronal data and their severe inter-subject variability. In this paper we extract efficient, personalized affective representations from EEG signals during music listening. To this end, we employ music signals as a supervisory modality to EEG, aiming to project their semantic correspondence onto a common representation space. We utilize a bi-modal framework by combining an LSTM-based attention model to process EEG and a pre-trained model for music tagging, along with a reverse domain discriminator to align the distributions of the two modalities, further constraining the learning process with emotion tags. The resulting framework can be utilized for emotion recognition both directly, by performing supervised predictions from either modality, and indirectly, by providing relevant music samples to EEG input queries. The experimental findings show the potential of enhancing neuronal data through stimulus information for recognition purposes and yield insights into the distribution and temporal variance of music-induced affective features.