# Attentive Cross-modal Connections for Deep Multimodal Wearable-based Emotion Recognition

arXiv: https://arxiv.org/abs/2108.02241 · HTML: https://mlsanity.com/paper/2108.02241/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 4 Aug 2021.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Self-supervised ECG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2002.03898/) (2020)
- [Self-supervised Learning for ECG-based Emotion Recognition](https://mlsanity.com/paper/1910.07497/) (2019)
- [Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing](https://mlsanity.com/paper/1905.07039/) (2019)
- [Classification of Hand Movements from EEG using a Deep Attention-based LSTM Network](https://mlsanity.com/paper/1908.02252/) (2019)
- [XFlow: Cross-modal Deep Neural Networks for Audiovisual Classification](https://mlsanity.com/paper/1709.00572/) (2017)
- [Unsupervised Multi-Modal Representation Learning for Affective Computing with Multi-Corpus Wearable Data](https://mlsanity.com/paper/2008.10726/) (2020)
- [X-CNN: Cross-modal Convolutional Neural Networks for Sparse Datasets](https://mlsanity.com/paper/1610.00163/) (2016)
- [Classification of Cognitive Load and Expertise for Adaptive Simulation using Deep Multitask Learning](https://mlsanity.com/paper/1908.00385/) (2019)

## Built on by (most cited first, 9 of 9)

- [PARSE: Pairwise Alignment of Representations in Semi-Supervised EEG Learning for Emotion Recognition](https://mlsanity.com/paper/2202.05400/) (2022)
- [CLARE: Cognitive Load Assessment in REaltime with Multimodal Data](https://mlsanity.com/paper/2404.17098/) (2024)
- [AttX: Attentive Cross-Connections for Fusion of Wearable Signals in Emotion Recognition](https://mlsanity.com/paper/2206.04625/) (2022)
- [Context-sensitive neocortical neurons transform the effectiveness and efficiency of neural information processing](https://mlsanity.com/paper/2207.07338/) (2022)
- [Unlocking the potential of two-point cells for energy-efficient and resilient training of deep nets](https://mlsanity.com/paper/2211.01950/) (2022)
- [Vital Sign Forecasting for Sepsis Patients in ICUs](https://mlsanity.com/paper/2311.04770/) (2023)
- [Partial Label Learning for Emotion Recognition from EEG](https://mlsanity.com/paper/2302.13170/) (2023)
- [SelfReplay: Adapting Self-Supervised Sensory Models via Adaptive Meta-Task Replay](https://mlsanity.com/paper/2404.15305/) (2024)
- [Tracing Human Stress from Physiological Signals using UWB Radar](https://mlsanity.com/paper/2410.10155/) (2024)

## Abstract (arXiv)

Classification of human emotions can play an essential role in the design and improvement of human-machine systems. While individual biological signals such as Electrocardiogram (ECG) and Electrodermal Activity (EDA) have been widely used for emotion recognition with machine learning methods, multimodal approaches generally fuse extracted features or final classification/regression results to boost performance. To enhance multimodal learning, we present a novel attentive cross-modal connection to share information between convolutional neural networks responsible for learning individual modalities. Specifically, these connections improve emotion classification by sharing intermediate representations among EDA and ECG and apply attention weights to the shared information, thus learning more effective multimodal embeddings. We perform experiments on the WESAD dataset to identify the best configuration of the proposed method for emotion classification. Our experiments show that the proposed approach is capable of learning strong multimodal representations and outperforms a number of baselines methods.