# A Multimodal Intermediate Fusion Network with Manifold Learning for Stress Detection

arXiv: https://arxiv.org/abs/2403.08077 · HTML: https://mlsanity.com/paper/2403.08077/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 39 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Builds on (most cited first, 8 of 8)

- [A Survey on Model Compression for Large Language Models](https://mlsanity.com/paper/2308.07633/) (2023)
- [A multimodal sensor dataset for continuous stress detection of nurses in a hospital](https://mlsanity.com/paper/2108.07689/) (2021)
- [Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion with Missing Data](https://mlsanity.com/paper/2210.17311/) (2022)
- [Employing Multimodal Machine Learning for Stress Detection](https://mlsanity.com/paper/2306.09385/) (2023)
- [A multimodal stress detection dataset with facial expressions and physiological signals](https://mlsanity.com/paper/2209.13542/) (2022)
- [Regularization and Optimization strategies in Deep Convolutional Neural Network](https://mlsanity.com/paper/1712.04711/) (2017)
- [Fusion of Physiological and Behavioural Signals on SPD Manifolds with Application to Stress and Pain Detection](https://mlsanity.com/paper/2207.08811/) (2022)
- [Multimodal Stress Detection Using Facial Landmarks and Biometric Signals](https://mlsanity.com/paper/2311.03606/) (2023)

## Built on by (most cited first, 5 of 5)

- [Hierarchical SegNet with Channel and Context Attention for Accurate Lung Segmentation in Chest X-ray Images](https://mlsanity.com/paper/2405.12318/) (2024)
- [In the Search for Optimal Multi-view Learning Models for Crop Classification with Global Remote Sensing Data](https://mlsanity.com/paper/2403.16582/) (2024)
- [Exploring Eye Tracking to Detect Cognitive Load in Complex Virtual Reality Training](https://mlsanity.com/paper/2411.12771/) (2024)
- [UL-DD: A Multimodal Drowsiness Dataset Using Video, Biometric Signals, and Behavioral Data](https://mlsanity.com/paper/2507.13403/) (2025)
- [Learning Reconstructive Embeddings in Reproducing Kernel Hilbert Spaces via the Representer Theorem](https://mlsanity.com/paper/2601.05811/) (2026)

## Abstract (arXiv)

Multimodal deep learning methods capture synergistic features from multiple modalities and have the potential to improve accuracy for stress detection compared to unimodal methods. However, this accuracy gain typically comes from high computational cost due to the high-dimensional feature spaces, especially for intermediate fusion. Dimensionality reduction is one way to optimize multimodal learning by simplifying data and making the features more amenable to processing and analysis, thereby reducing computational complexity. This paper introduces an intermediate multimodal fusion network with manifold learning-based dimensionality reduction. The multimodal network generates independent representations from biometric signals and facial landmarks through 1D-CNN and 2D-CNN. Finally, these features are fused and fed to another 1D-CNN layer, followed by a fully connected dense layer. We compared various dimensionality reduction techniques for different variations of unimodal and multimodal networks. We observe that the intermediate-level fusion with the Multi-Dimensional Scaling (MDS) manifold method showed promising results with an accuracy of 96.00% in a Leave-One-Subject-Out Cross-Validation (LOSO-CV) paradigm over other dimensional reduction methods. MDS had the highest computational cost among manifold learning methods. However, while outperforming other networks, it managed to reduce the computational cost of the proposed networks by 25% when compared to six well-known conventional feature selection methods used in the preprocessing step.