# Interpretable Multimodal Emotion Recognition using Facial Features and Physiological Signals

arXiv: https://arxiv.org/abs/2306.02845 · HTML: https://mlsanity.com/paper/2306.02845/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 8 of 8)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [Learning Multi-level Deep Representations for Image Emotion Classification](https://mlsanity.com/paper/1611.07145/) (2016)
- [Facial Feature Point Detection: A Comprehensive Survey](https://mlsanity.com/paper/1410.1037/) (2014)

## Built on by (most cited first, 2 of 2)

- [Generalization of Video-Based Heart Rate Estimation Methods To Low Illumination and Elevated Heart Rates](https://mlsanity.com/paper/2503.11697/) (2025)
- [VisioPhysioENet: Visual Physiological Engagement Detection Network](https://mlsanity.com/paper/2409.16126/) (2024)

## Abstract (arXiv)

This paper aims to demonstrate the importance and feasibility of fusing multimodal information for emotion recognition. It introduces a multimodal framework for emotion understanding by fusing the information from visual facial features and rPPG signals extracted from the input videos. An interpretability technique based on permutation feature importance analysis has also been implemented to compute the contributions of rPPG and visual modalities toward classifying a given input video into a particular emotion class. The experiments on IEMOCAP dataset demonstrate that the emotion classification performance improves by combining the complementary information from multiple modalities.