# Multimodal Stress Detection Using Facial Landmarks and Biometric Signals

arXiv: https://arxiv.org/abs/2311.03606 · HTML: https://mlsanity.com/paper/2311.03606/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 67 references that are in the map.
- First posted on arXiv 6 Nov 2023.

## Builds on (most cited first, 7 of 7)

- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [A multimodal sensor dataset for continuous stress detection of nurses in a hospital](https://mlsanity.com/paper/2108.07689/) (2021)
- [Automatic Stress Detection in Working Environments from Smartphones' Accelerometer Data: A First Step](https://mlsanity.com/paper/1510.04221/) (2015)
- [Multi-Context Dual Hyper-Prior Neural Image Compression](https://mlsanity.com/paper/2309.10799/) (2023)
- [Employing Multimodal Machine Learning for Stress Detection](https://mlsanity.com/paper/2306.09385/) (2023)
- [A multimodal stress detection dataset with facial expressions and physiological signals](https://mlsanity.com/paper/2209.13542/) (2022)
- [Personalized Prediction of Recurrent Stress Events Using Self-Supervised Learning on Multimodal Time-Series Data](https://mlsanity.com/paper/2307.03337/) (2023)

## Built on by (most cited first, 2 of 2)

- [A Multimodal Intermediate Fusion Network with Manifold Learning for Stress Detection](https://mlsanity.com/paper/2403.08077/) (2024)
- [UL-DD: A Multimodal Drowsiness Dataset Using Video, Biometric Signals, and Behavioral Data](https://mlsanity.com/paper/2507.13403/) (2025)

## Abstract (arXiv)

The development of various sensing technologies is improving measurements of stress and the well-being of individuals. Although progress has been made with single signal modalities like wearables and facial emotion recognition, integrating multiple modalities provides a more comprehensive understanding of stress, given that stress manifests differently across different people. Multi-modal learning aims to capitalize on the strength of each modality rather than relying on a single signal. Given the complexity of processing and integrating high-dimensional data from limited subjects, more research is needed. Numerous research efforts have been focused on fusing stress and emotion signals at an early stage, e.g., feature-level fusion using basic machine learning methods and 1D-CNN Methods. This paper proposes a multi-modal learning approach for stress detection that integrates facial landmarks and biometric signals. We test this multi-modal integration with various early-fusion and late-fusion techniques to integrate the 1D-CNN model from biometric signals and 2-D CNN using facial landmarks. We evaluate these architectures using a rigorous test of models' generalizability using the leave-one-subject-out mechanism, i.e., all samples related to a single subject are left out to train the model. Our findings show that late-fusion achieved 94.39% accuracy, and early-fusion surpassed it with a 98.38% accuracy rate. This research contributes valuable insights into enhancing stress detection through a multi-modal approach. The proposed research offers important knowledge in improving stress detection using a multi-modal approach.