mlsanityData to 14 Oct 2026

MVP: Multimodal Emotion Recognition based on Video and Physiological Signals

arXiv 2501.03103 ↗First posted 6 Jan 2025cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Human emotions entail a complex set of behavioral, physiological and cognitive changes. Current state-of-the-art models fuse the behavioral and physiological components using classic machine learning, rather than recent deep learning techniques. We propose to fill this gap, designing the Multimodal for Video and Physio (MVP) architecture, streamlined to fuse video and physiological signals. Differently then others approaches, MVP exploits the benefits of attention to enable the use of long input sequences (1-2 minutes). We have studied video and physiological backbones for inputting long sequences and evaluated our method with respect to the state-of-the-art. Our results show that MVP outperforms former methods for emotion recognition based on facial videos, EDA, and ECG/PPG.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 14 of its 34 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

14 of 34 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  3. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  4. 2022MARLIN: Masked Autoencoder for facial video Representation LearnINgcited by 57
  5. 2022Transformer-based Multimodal Information Fusion for Facial Expression Analysiscited by 49
  6. 2020A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysiscited by 46
  7. 2021Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysiscited by 45
  8. 2019Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computingcited by 22
  9. 2022M-SENA: An Integrated Platform for Multimodal Sentiment Analysiscited by 21
  10. 2022Transformer-Based Self-Supervised Learning for Emotion Recognitioncited by 16
  11. 2021HetEmotionNet: Two-Stream Heterogeneous Graph Recurrent Neural Network for Multi-modal Emotion Recognitioncited by 15
  12. 2020Unsupervised Multi-Modal Representation Learning for Affective Computing with Multi-Corpus Wearable Datacited by 10
2 more
  1. 2022Multi-Task Learning Framework for Emotion Recognition in-the-wildcited by 10
  2. 2022Emotion Recognition with Pre-Trained Transformers Using Multimodal Signalscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026BioKD: Selective Physiology-to-Video Knowledge Distillation via Reliability Gate for Emotion Recognitioncited by 0
  2. 2026A Comprehensive Review of Multimodal Facial State Analysis: Tasks, Methods, and Resourcescited by 0