# Multi-modal Mood Reader: Pre-trained Model Empowers Cross-Subject Emotion Recognition

arXiv: https://arxiv.org/abs/2405.19373 · HTML: https://mlsanity.com/paper/2405.19373/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 48 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 18 of 18)

- [Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding](https://mlsanity.com/paper/2211.06956/) (2022)
- [EEG-Based Emotion Recognition Using Regularized Graph Neural Networks](https://mlsanity.com/paper/1907.07835/) (2019)
- [TransVOD: End-to-End Video Object Detection with Spatial-Temporal Transformers](https://mlsanity.com/paper/2201.05047/) (2022)
- [Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity](https://mlsanity.com/paper/2305.11675/) (2023)
- [Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition](https://mlsanity.com/paper/2109.09559/) (2021)
- [DreamDiffusion: Generating High-Quality Images from Brain EEG Signals](https://mlsanity.com/paper/2306.16934/) (2023)
- [STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition](https://mlsanity.com/paper/2210.07503/) (2022)
- [High-Resolution Document Shadow Removal via A Large-Scale Real-World Dataset and A Frequency-Aware Shadow Erasing Net](https://mlsanity.com/paper/2308.14221/) (2023)
- [Fine Perceptive GANs for Brain MR Image Super-Resolution in Wavelet Domain](https://mlsanity.com/paper/2011.04145/) (2020)
- [A Large-scale Film Style Dataset for Learning Multi-frequency Driven Film Enhancement](https://mlsanity.com/paper/2301.08880/) (2023)
- [Multimodal Representations Learning and Adversarial Hypergraph Fusion for Early Alzheimer's Disease Prediction](https://mlsanity.com/paper/2107.09928/) (2021)
- [Characterization Multimodal Connectivity of Brain Network by Hypergraph GAN for Alzheimer's Disease Analysis](https://mlsanity.com/paper/2107.09953/) (2021)
- [Devignet: High-Resolution Vignetting Removal via a Dual Aggregated Fusion Transformer With Adaptive Channel Expansion](https://mlsanity.com/paper/2308.13739/) (2023)
- [HetEmotionNet: Two-Stream Heterogeneous Graph Recurrent Neural Network for Multi-modal Emotion Recognition](https://mlsanity.com/paper/2108.03354/) (2021)
- [GMSS: Graph-Based Multi-Task Self-Supervised Learning for EEG Emotion Recognition](https://mlsanity.com/paper/2205.01030/) (2022)
- [3D Brain Reconstruction by Hierarchical Shape-Perception Network from a Single Incomplete Image](https://mlsanity.com/paper/2107.11010/) (2021)
- [Spatio-Temporal Ranked-Attention Networks for Video Captioning](https://mlsanity.com/paper/2001.06127/) (2020)
- [Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals](https://mlsanity.com/paper/2212.13885/) (2022)

## Built on by (most cited first, 4 of 4)

- [Test-Time Intensity Consistency Adaptation for Shadow Detection](https://mlsanity.com/paper/2410.07695/) (2024)
- [High-Fidelity Functional Ultrasound Reconstruction via A Visual Auto-Regressive Framework](https://mlsanity.com/paper/2505.21530/) (2025)
- [ConnectomeDiffuser: Generative AI Enables Brain Network Construction from Diffusion Tensor Imaging](https://mlsanity.com/paper/2505.22683/) (2025)
- [Brain Network Analysis Based on Fine-tuned Self-supervised Model for Brain Disease Diagnosis](https://mlsanity.com/paper/2506.11671/) (2025)

## Abstract (arXiv)

Emotion recognition based on Electroencephalography (EEG) has gained significant attention and diversified development in fields such as neural signal processing and affective computing. However, the unique brain anatomy of individuals leads to non-negligible natural differences in EEG signals across subjects, posing challenges for cross-subject emotion recognition. While recent studies have attempted to address these issues, they still face limitations in practical effectiveness and model framework unity. Current methods often struggle to capture the complex spatial-temporal dynamics of EEG signals and fail to effectively integrate multimodal information, resulting in suboptimal performance and limited generalizability across subjects. To overcome these limitations, we develop a Pre-trained model based Multimodal Mood Reader for cross-subject emotion recognition that utilizes masked brain signal modeling and interlinked spatial-temporal attention mechanism. The model learns universal latent representations of EEG signals through pre-training on large scale dataset, and employs Interlinked spatial-temporal attention mechanism to process Differential Entropy(DE) features extracted from EEG data. Subsequently, a multi-level fusion layer is proposed to integrate the discriminative features, maximizing the advantages of features across different dimensions and modalities. Extensive experiments on public datasets demonstrate Mood Reader's superior performance in cross-subject emotion recognition tasks, outperforming state-of-the-art methods. Additionally, the model is dissected from attention perspective, providing qualitative analysis of emotion-related brain areas, offering valuable insights for affective research in neural signal processing.