# Multi-level Stress Assessment from ECG in a Virtual Reality Environment using Multimodal Fusion

arXiv: https://arxiv.org/abs/2107.04566 · HTML: https://mlsanity.com/paper/2107.04566/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 54 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Self-supervised ECG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2002.03898/) (2020)
- [Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science](https://mlsanity.com/paper/1603.06212/) (2016)
- [A dataset of continuous affect annotations and physiological signals for emotion analysis](https://mlsanity.com/paper/1812.02782/) (2018)
- [Autostacker: A Compositional Evolutionary Learning System](https://mlsanity.com/paper/1803.00684/) (2018)
- [CNN based Multistage Gated Average Fusion (MGAF) for Human Action Recognition Using Depth and Inertial Sensors](https://mlsanity.com/paper/2010.16073/) (2020)
- [Human Action Recognition Using Deep Multilevel Multimodal (M2) Fusion of Depth and Inertial Sensors](https://mlsanity.com/paper/1910.11482/) (2019)
- [Multi-level Stress Assessment Using Multi-domain Fusion of ECG Signal](https://mlsanity.com/paper/2008.05503/) (2020)
- [ECG Heart-beat Classification Using Multimodal Image Fusion](https://mlsanity.com/paper/2105.13536/) (2021)
- [Towards Improved Human Action Recognition Using Convolutional Neural Networks and Multimodal Fusion of Depth and Inertial Sensor Data](https://mlsanity.com/paper/2008.09747/) (2020)

## Built on by (most cited first, 2 of 2)

- [Stress Classification from ECG Signals Using Vision Transformer](https://mlsanity.com/paper/2603.26721/) (2026)
- [Stress Detection from Photoplethysmography in a Virtual Reality Environment](https://mlsanity.com/paper/2409.17427/) (2024)

## Abstract (arXiv)

ECG is an attractive option to assess stress in serious Virtual Reality (VR) applications due to its non-invasive nature. However, the existing Machine Learning (ML) models perform poorly. Moreover, existing studies only perform a binary stress assessment, while to develop a more engaging biofeedback-based application, multi-level assessment is necessary. Existing studies annotate and classify a single experience (e.g. watching a VR video) to a single stress level, which again prevents design of dynamic experiences where real-time in-game stress assessment can be utilized. In this paper, we report our findings on a new study on VR stress assessment, where three stress levels are assessed. ECG data was collected from 9 users experiencing a VR roller coaster. The VR experience was then manually labeled in 10-seconds segments to three stress levels by three raters. We then propose a novel multimodal deep fusion model utilizing spectrogram and 1D ECG that can provide a stress prediction from just a 1-second window. Experimental results demonstrate that the proposed model outperforms the classical HRV-based ML models (9% increase in accuracy) and baseline deep learning models (2.5% increase in accuracy). We also report results on the benchmark WESAD dataset to show the supremacy of the model.