# The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress

arXiv: https://arxiv.org/abs/2207.05691 · HTML: https://mlsanity.com/paper/2207.05691/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 60 references that are in the map.
- First posted on arXiv 23 Jun 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)
- [The INTERSPEECH 2021 Computational Paralinguistics Challenge: COVID-19 Cough, COVID-19 Speech, Escalation & Primates](https://mlsanity.com/paper/2102.13468/) (2021)
- [The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress](https://mlsanity.com/paper/2104.07123/) (2021)
- [The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and Improvements](https://mlsanity.com/paper/2101.06053/) (2021)
- [EmoNet: A Transfer Learning Framework for Multi-Corpus Speech Emotion Recognition](https://mlsanity.com/paper/2103.08310/) (2021)
- [End2You -- The Imperial Toolkit for Multimodal Profiling by End-to-End Learning](https://mlsanity.com/paper/1802.01115/) (2018)
- [DeepSpectrumLite: A Power-Efficient Transfer Learning Framework for Embedded Speech and Audio Processing from Decentralised Data](https://mlsanity.com/paper/2104.11629/) (2021)
- ["So You Think You're Funny?": Rating the Humour Quotient in Standup Comedy](https://mlsanity.com/paper/2110.12765/) (2021)
- [MuSe-Toolbox: The Multimodal Sentiment Analysis Continuous Annotation Fusion and Discrete Class Transformation Toolbox](https://mlsanity.com/paper/2107.11757/) (2021)
- [Hybrid Mutimodal Fusion for Dimensional Emotion Recognition](https://mlsanity.com/paper/2110.08495/) (2021)

## Built on by (most cited first, 23 of 23)

- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [MMA-MRNNet: Harnessing Multiple Models of Affect and Dynamic Masked RNN for Precise Facial Expression Intensity Estimation](https://mlsanity.com/paper/2303.00180/) (2023)
- [Hybrid Multimodal Feature Extraction, Mining and Fusion for Sentiment Analysis](https://mlsanity.com/paper/2208.03051/) (2022)
- [The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation](https://mlsanity.com/paper/2305.03369/) (2023)
- [Hybrid Multimodal Fusion for Humor Detection](https://mlsanity.com/paper/2209.11949/) (2022)
- [Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges](https://mlsanity.com/paper/2407.16804/) (2024)
- [Systematic Literature Review: Computational Approaches for Humour Style Classification](https://mlsanity.com/paper/2402.01759/) (2024)
- [A Two-Model Approach for Humour Style Recognition](https://mlsanity.com/paper/2410.12842/) (2024)
- [Multimodal Feature Extraction and Fusion for Emotional Reaction Intensity Estimation and Expression Classification in Videos with Transformers](https://mlsanity.com/paper/2303.09164/) (2023)
- [Computational Analysis of Stress, Depression and Engagement in Mental Health: A Survey](https://mlsanity.com/paper/2403.08824/) (2024)
- [Towards Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results](https://mlsanity.com/paper/2209.14272/) (2022)
- [Modeling Emotional Trajectories in Written Stories Utilizing Transformers and Weakly-Supervised Learning](https://mlsanity.com/paper/2406.02251/) (2024)
- [Computer Vision Estimation of Emotion Reaction Intensity in the Wild](https://mlsanity.com/paper/2303.10741/) (2023)
- [On the Wings of Imagination: Conflicting Script-based Multi-role Framework for Humor Caption Generation](https://mlsanity.com/paper/2602.06423/) (2026)
- [A Dual Branch Network for Emotional Reaction Intensity Estimation](https://mlsanity.com/paper/2303.09210/) (2023)
- [Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognition](https://mlsanity.com/paper/2311.10119/) (2023)
- [Predicting Heart Activity from Speech using Data-driven and Knowledge-based features](https://mlsanity.com/paper/2406.06341/) (2024)
- [Automatic Emotion Modelling in Written Stories](https://mlsanity.com/paper/2212.11382/) (2022)
- [Human Reaction Intensity Estimation with Ensemble of Multi-task Networks](https://mlsanity.com/paper/2303.09240/) (2023)
- [Integrating Holistic and Local Information to Estimate Emotional Reaction Intensity](https://mlsanity.com/paper/2305.05534/) (2023)
- [Computational Charisma -- A Brick by Brick Blueprint for Building Charismatic Artificial Intelligence](https://mlsanity.com/paper/2301.00142/) (2022)
- [Exploiting Diverse Feature for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2308.13421/) (2023)
- [To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition](https://mlsanity.com/paper/2605.04877/) (2026)

## Abstract (arXiv)

The Multimodal Sentiment Analysis Challenge (MuSe) 2022 is dedicated to multimodal sentiment and emotion recognition. For this year's challenge, we feature three datasets: (i) the Passau Spontaneous Football Coach Humor (Passau-SFCH) dataset that contains audio-visual recordings of German football coaches, labelled for the presence of humour; (ii) the Hume-Reaction dataset in which reactions of individuals to emotional stimuli have been annotated with respect to seven emotional expression intensities, and (iii) the Ulm-Trier Social Stress Test (Ulm-TSST) dataset comprising of audio-visual data labelled with continuous emotion values (arousal and valence) of people in stressful dispositions. Using the introduced datasets, MuSe 2022 2022 addresses three contemporary affective computing problems: in the Humor Detection Sub-Challenge (MuSe-Humor), spontaneous humour has to be recognised; in the Emotional Reactions Sub-Challenge (MuSe-Reaction), seven fine-grained `in-the-wild' emotions have to be predicted; and in the Emotional Stress Sub-Challenge (MuSe-Stress), a continuous prediction of stressed emotion values is featured. The challenge is designed to attract different research communities, encouraging a fusion of their disciplines. Mainly, MuSe 2022 targets the communities of audio-visual emotion recognition, health informatics, and symbolic sentiment analysis. This baseline paper describes the datasets as well as the feature sets extracted from them. A recurrent neural network with LSTM cells is used to set competitive baseline results on the test partitions for each sub-challenge. We report an Area Under the Curve (AUC) of .8480 for MuSe-Humor; .2801 mean (from 7-classes) Pearson's Correlations Coefficient for MuSe-Reaction, as well as .4931 Concordance Correlation Coefficient (CCC) and .4761 for valence and arousal in MuSe-Stress, respectively.