# HetEmotionNet: Two-Stream Heterogeneous Graph Recurrent Neural Network for Multi-modal Emotion Recognition

arXiv: https://arxiv.org/abs/2108.03354 · HTML: https://mlsanity.com/paper/2108.03354/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 62 references that are in the map.
- First posted on arXiv 7 Aug 2021.

## Builds on (most cited first, 14 of 14)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [Graph Transformer Networks](https://mlsanity.com/paper/1911.06455/) (2019)
- [M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cues](https://mlsanity.com/paper/1911.05659/) (2019)
- [Identifying Stable Patterns over Time for Emotion Recognition from EEG](https://mlsanity.com/paper/1601.02197/) (2016)
- [Spatial-Temporal Recurrent Neural Network for Emotion Recognition](https://mlsanity.com/paper/1705.04515/) (2017)
- [SalientSleepNet: Multimodal Salient Wave Detection Network for Sleep Staging](https://mlsanity.com/paper/2105.13864/) (2021)
- [Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis](https://mlsanity.com/paper/1908.05349/) (2019)
- [Multimodal Emotion Recognition Using Multimodal Deep Learning](https://mlsanity.com/paper/1602.08225/) (2016)
- [Detecting causality in multivariate time series via non-uniform embedding](https://mlsanity.com/paper/1903.05842/) (2019)

## Built on by (most cited first, 15 of 15)

- [Multi-View Spatial-Temporal Graph Convolutional Networks with Domain Generalization for Sleep Stage Classification](https://mlsanity.com/paper/2109.01824/) (2021)
- [EEG-Deformer: A Dense Convolutional Transformer for Brain-computer Interfaces](https://mlsanity.com/paper/2405.00719/) (2024)
- [Graph Neural Networks in EEG-based Emotion Recognition: A Survey](https://mlsanity.com/paper/2402.01138/) (2024)
- [GNN4EEG: A Benchmark and Toolkit for Electroencephalography Classification with Graph Neural Network](https://mlsanity.com/paper/2309.15515/) (2023)
- [Multi-modal Mood Reader: Pre-trained Model Empowers Cross-Subject Emotion Recognition](https://mlsanity.com/paper/2405.19373/) (2024)
- [A Comprehensive Survey on EEG-Based Emotion Recognition: A Graph-Based Perspective](https://mlsanity.com/paper/2408.06027/) (2024)
- [A Survey of Spatio-Temporal EEG data Analysis: from Models to Applications](https://mlsanity.com/paper/2410.08224/) (2024)
- [MVP: Multimodal Emotion Recognition based on Video and Physiological Signals](https://mlsanity.com/paper/2501.03103/) (2025)
- [CIT-EmotionNet: CNN Interactive Transformer Network for EEG Emotion Recognition](https://mlsanity.com/paper/2305.05548/) (2023)
- [EEGDM: Learning EEG Representation with Latent Diffusion Model](https://mlsanity.com/paper/2508.20705/) (2025)
- [TEn-CATG:Text-Enriched Audio-Visual Video Parsing with Multi-Scale Category-Aware Temporal Graph](https://mlsanity.com/paper/2509.04086/) (2025)
- [H2G2-Net: A Hierarchical Heterogeneous Graph Generative Network Framework for Discovery of Multi-Modal Physiological Responses](https://mlsanity.com/paper/2401.02905/) (2024)
- [LibEMER: A novel benchmark and algorithms library for EEG-based Multimodal Emotion Recognition](https://mlsanity.com/paper/2509.19330/) (2025)
- [Adaptive Temporal Dynamics for Personalized Emotion Recognition: A Liquid Neural Network Approach](https://mlsanity.com/paper/2602.06997/) (2026)
- [Multimodal Wearable-Based Olfactory-Induced Emotion Recognition in Arousal-Valence Dimensions](https://mlsanity.com/paper/2608.00043/) (2026)

## Abstract (arXiv)

The research on human emotion under multimedia stimulation based on physiological signals is an emerging field, and important progress has been achieved for emotion recognition based on multi-modal signals. However, it is challenging to make full use of the complementarity among spatial-spectral-temporal domain features for emotion recognition, as well as model the heterogeneity and correlation among multi-modal signals. In this paper, we propose a novel two-stream heterogeneous graph recurrent neural network, named HetEmotionNet, fusing multi-modal physiological signals for emotion recognition. Specifically, HetEmotionNet consists of the spatial-temporal stream and the spatial-spectral stream, which can fuse spatial-spectral-temporal domain features in a unified framework. Each stream is composed of the graph transformer network for modeling the heterogeneity, the graph convolutional network for modeling the correlation, and the gated recurrent unit for capturing the temporal domain or spectral domain dependency. Extensive experiments on two real-world datasets demonstrate that our proposed model achieves better performance than state-of-the-art baselines.