# MUSER: MUltimodal Stress Detection using Emotion Recognition as an Auxiliary Task

arXiv: https://arxiv.org/abs/2105.08146 · HTML: https://mlsanity.com/paper/2105.08146/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 49 references that are in the map.
- First posted on arXiv 17 May 2021.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [REMIND Your Neural Network to Prevent Catastrophic Forgetting](https://mlsanity.com/paper/1910.02509/) (2019)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [EmotionLines: An Emotion Corpus of Multi-Party Conversations](https://mlsanity.com/paper/1802.08379/) (2018)
- [The OMG-Emotion Behavior Dataset](https://mlsanity.com/paper/1803.05434/) (2018)
- [Dynamic Multi-Level Multi-Task Learning for Sentence Simplification](https://mlsanity.com/paper/1806.07304/) (2018)
- [Multi-Task Video Captioning with Video and Entailment Generation](https://mlsanity.com/paper/1704.07489/) (2017)
- [MuSE-ing on the Impact of Utterance Ordering On Crowdsourced Emotion Annotations](https://mlsanity.com/paper/1903.11672/) (2019)

## Built on by (most cited first, 4 of 4)

- [Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges](https://mlsanity.com/paper/2407.16804/) (2024)
- [Expanding the Role of Affective Phenomena in Multimodal Interaction Research](https://mlsanity.com/paper/2305.10827/) (2023)
- [Dynamic Stress Detection: A Study of Temporal Progression Modelling of Stress in Speech](https://mlsanity.com/paper/2510.08586/) (2025)
- [Efficient-Husformer: Efficient Multimodal Transformer Hyperparameter Optimization for Stress and Cognitive Loads](https://mlsanity.com/paper/2511.22362/) (2025)

## Abstract (arXiv)

The capability to automatically detect human stress can benefit artificial intelligent agents involved in affective computing and human-computer interaction. Stress and emotion are both human affective states, and stress has proven to have important implications on the regulation and expression of emotion. Although a series of methods have been established for multimodal stress detection, limited steps have been taken to explore the underlying inter-dependence between stress and emotion. In this work, we investigate the value of emotion recognition as an auxiliary task to improve stress detection. We propose MUSER -- a transformer-based model architecture and a novel multi-task learning algorithm with speed-based dynamic sampling strategy. Evaluations on the Multimodal Stressed Emotion (MuSE) dataset show that our model is effective for stress detection with both internal and external auxiliary tasks, and achieves state-of-the-art results.