# Hierarchical Hypercomplex Network for Multimodal Emotion Recognition

arXiv: https://arxiv.org/abs/2409.09194 · HTML: https://mlsanity.com/paper/2409.09194/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 26 references that are in the map.
- First posted on arXiv 13 Sep 2024.

## Builds on (most cited first, 6 of 6)

- [Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters](https://mlsanity.com/paper/2102.08597/) (2021)
- [PHNNs: Lightweight Neural Networks via Parameterized Hypercomplex Convolutions](https://mlsanity.com/paper/2110.04176/) (2021)
- [Hypercomplex Multimodal Emotion Recognition from EEG and Peripheral Physiological Signals](https://mlsanity.com/paper/2310.07648/) (2023)
- [Multi-View Hypercomplex Learning for Breast Cancer Screening](https://mlsanity.com/paper/2204.05798/) (2022)
- [Attention-Map Augmentation for Hypercomplex Breast Cancer Classification](https://mlsanity.com/paper/2310.07633/) (2023)
- [Towards Explaining Hypercomplex Neural Networks](https://mlsanity.com/paper/2403.17929/) (2024)

## Built on by (most cited first, 4 of 4)

- [Guess What I Think: Streamlined EEG-to-Image Generation with Latent Diffusion Models](https://mlsanity.com/paper/2410.02780/) (2024)
- [Cross-Temporal Attention Fusion (CTAF) for Multimodal Physiological Signals in Self-Supervised Learning](https://mlsanity.com/paper/2602.02784/) (2026)
- [Multimodal Functional Maximum Correlation for Emotion Recognition](https://mlsanity.com/paper/2512.23076/) (2025)
- [Multimodal Wearable-Based Olfactory-Induced Emotion Recognition in Arousal-Valence Dimensions](https://mlsanity.com/paper/2608.00043/) (2026)

## Abstract (arXiv)

Emotion recognition is relevant in various domains, ranging from healthcare to human-computer interaction. Physiological signals, being beyond voluntary control, offer reliable information for this purpose, unlike speech and facial expressions which can be controlled at will. They reflect genuine emotional responses, devoid of conscious manipulation, thereby enhancing the credibility of emotion recognition systems. Nonetheless, multimodal emotion recognition with deep learning models remains a relatively unexplored field. In this paper, we introduce a fully hypercomplex network with a hierarchical learning structure to fully capture correlations. Specifically, at the encoder level, the model learns intra-modal relations among the different channels of each input signal. Then, a hypercomplex fusion module learns inter-modal relations among the embeddings of the different modalities. The main novelty is in exploiting intra-modal relations by endowing the encoders with parameterized hypercomplex convolutions (PHCs) that thanks to hypercomplex algebra can capture inter-channel interactions within single modalities. Instead, the fusion module comprises parameterized hypercomplex multiplications (PHMs) that can model inter-modal correlations. The proposed architecture surpasses state-of-the-art models on the MAHNOB-HCI dataset for emotion recognition, specifically in classifying valence and arousal from electroencephalograms (EEGs) and peripheral physiological signals. The code of this study is available at https://github.com/ispamm/MHyEEG.