# Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing

arXiv: https://arxiv.org/abs/1905.07039 · HTML: https://mlsanity.com/paper/1905.07039/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2019.

## Built on by (most cited first, 22 of 22)

- [Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition](https://mlsanity.com/paper/2109.09559/) (2021)
- [LGGNet: Learning from Local-Global-Graph Representations for Brain-Computer Interface](https://mlsanity.com/paper/2105.02786/) (2021)
- [Transformer-Based Self-Supervised Learning for Emotion Recognition](https://mlsanity.com/paper/2204.05103/) (2022)
- [Refashioning Emotion Recognition Modelling: The Advent of Generalised Large Models](https://mlsanity.com/paper/2308.11578/) (2023)
- [Attentive Cross-modal Connections for Deep Multimodal Wearable-based Emotion Recognition](https://mlsanity.com/paper/2108.02241/) (2021)
- [Mental State Classification Using Multi-graph Features](https://mlsanity.com/paper/2203.00516/) (2022)
- [AttX: Attentive Cross-Connections for Fusion of Wearable Signals in Emotion Recognition](https://mlsanity.com/paper/2206.04625/) (2022)
- [Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals](https://mlsanity.com/paper/2212.13885/) (2022)
- [A Hybrid End-to-End Spatio-Temporal Attention Neural Network with Graph-Smooth Signals for EEG Emotion Recognition](https://mlsanity.com/paper/2307.03068/) (2023)
- [Deep learning applied to EEG data with different montages using spatial attention](https://mlsanity.com/paper/2310.10550/) (2023)
- [Deep adaptative spectral zoom for improved remote heart rate estimation](https://mlsanity.com/paper/2403.06902/) (2024)
- [MVP: Multimodal Emotion Recognition based on Video and Physiological Signals](https://mlsanity.com/paper/2501.03103/) (2025)
- [Milmer: a Framework for Multiple Instance Learning based Multimodal Emotion Recognition](https://mlsanity.com/paper/2502.00547/) (2025)
- [Assessing learned features of Deep Learning applied to EEG](https://mlsanity.com/paper/2111.04309/) (2021)
- [VRMN-bD: A Multi-modal Natural Behavior Dataset of Immersive Human Fear Responses in VR Stand-up Interactive Games](https://mlsanity.com/paper/2401.12133/) (2024)
- [A Unified Transformer-based Network for multimodal Emotion Recognition](https://mlsanity.com/paper/2308.14160/) (2023)
- [Exact Fit Attention in Node-Holistic Graph Convolutional Network for Improved EEG-Based Driver Fatigue Detection](https://mlsanity.com/paper/2501.15062/) (2025)
- [CAST-Phys: Contactless Affective States Through Physiological signals Database](https://mlsanity.com/paper/2507.06080/) (2025)
- [Learning Domain- and Class-Disentangled Prototypes for Domain-Generalized EEG Emotion Recognition](https://mlsanity.com/paper/2509.01135/) (2025)
- [Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition](https://mlsanity.com/paper/2605.05694/) (2026)
- [Computer Vision Based Neurology Brain Activity Rejection Architecture and Implementation](https://mlsanity.com/paper/2607.21654/) (2026)
- [Take it Personally: The Limits of General SSL Representations for Real-Life PPG Emotion Detection](https://mlsanity.com/paper/2608.14675/) (2026)

## Abstract (arXiv)

In recent years, the use of bio-sensing signals such as electroencephalogram (EEG), electrocardiogram (ECG), etc. have garnered interest towards applications in affective computing. The parallel trend of deep-learning has led to a huge leap in performance towards solving various vision-based research problems such as object detection. Yet, these advances in deep-learning have not adequately translated into bio-sensing research. This work applies novel deep-learning-based methods to various bio-sensing and video data of four publicly available multi-modal emotion datasets. For each dataset, we first individually evaluate the emotion-classification performance obtained by each modality. We then evaluate the performance obtained by fusing the features from these modalities. We show that our algorithms outperform the results reported by other studies for emotion/valence/arousal/liking classification on DEAP and MAHNOB-HCI datasets and set up benchmarks for the newer AMIGOS and DREAMER datasets. We also evaluate the performance of our algorithms by combining the datasets and by using transfer learning to show that the proposed method overcomes the inconsistencies between the datasets. Hence, we do a thorough analysis of multi-modal affective data from more than 120 subjects and 2,800 trials. Finally, utilizing a convolution-deconvolution network, we propose a new technique towards identifying salient brain regions corresponding to various affective states.