# The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress

arXiv: https://arxiv.org/abs/2104.07123 · HTML: https://mlsanity.com/paper/2104.07123/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 56 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [Analysing Affective Behavior in the First ABAW 2020 Competition](https://mlsanity.com/paper/2001.11409/) (2020)
- [Exploring Hate Speech Detection in Multimodal Publications](https://mlsanity.com/paper/1910.03814/) (2019)
- [The INTERSPEECH 2021 Computational Paralinguistics Challenge: COVID-19 Cough, COVID-19 Speech, Escalation & Primates](https://mlsanity.com/paper/2102.13468/) (2021)
- [Sentiment Analysis: Automatically Detecting Valence, Emotions, and Other Affectual States from Text](https://mlsanity.com/paper/2005.11882/) (2020)
- [Cross-lingual Zero- and Few-shot Hate Speech Detection Utilising Frozen Transformer Language Models and AXEL](https://mlsanity.com/paper/2004.13850/) (2020)
- [The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and Improvements](https://mlsanity.com/paper/2101.06053/) (2021)
- [EMOPAIN Challenge 2020: Multimodal Pain Evaluation from Facial and Bodily Expressions](https://mlsanity.com/paper/2001.07739/) (2020)
- [The Many-to-Many Mapping Between the Concordance Correlation Coefficient and the Mean Square Error](https://mlsanity.com/paper/1902.05180/) (2019)

## Built on by (most cited first, 27 of 27)

- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning](https://mlsanity.com/paper/2304.08981/) (2023)
- [A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learning](https://mlsanity.com/paper/2312.05735/) (2023)
- [The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress](https://mlsanity.com/paper/2207.05691/) (2022)
- [COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition](https://mlsanity.com/paper/2206.05833/) (2022)
- [Probing Speech Emotion Recognition Transformers for Linguistic Knowledge](https://mlsanity.com/paper/2204.00400/) (2022)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation](https://mlsanity.com/paper/2305.03369/) (2023)
- [MuSe-Toolbox: The Multimodal Sentiment Analysis Continuous Annotation Fusion and Discrete Class Transformation Toolbox](https://mlsanity.com/paper/2107.11757/) (2021)
- [Multistage linguistic conditioning of convolutional layers for speech emotion recognition](https://mlsanity.com/paper/2110.06650/) (2021)
- [Hybrid Mutimodal Fusion for Dimensional Emotion Recognition](https://mlsanity.com/paper/2110.08495/) (2021)
- [Towards Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results](https://mlsanity.com/paper/2209.14272/) (2022)
- [Computational Analysis of Stress, Depression and Engagement in Mental Health: A Survey](https://mlsanity.com/paper/2403.08824/) (2024)
- [Modeling Emotional Trajectories in Written Stories Utilizing Transformers and Weakly-Supervised Learning](https://mlsanity.com/paper/2406.02251/) (2024)
- [EmoNet-Voice: A Fine-Grained, Expert-Verified Benchmark for Speech Emotion Detection](https://mlsanity.com/paper/2506.09827/) (2025)
- [Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognition](https://mlsanity.com/paper/2311.10119/) (2023)
- [Predicting Heart Activity from Speech using Data-driven and Knowledge-based features](https://mlsanity.com/paper/2406.06341/) (2024)
- [GraphTMT: Unsupervised Graph-based Topic Modeling from Video Transcripts](https://mlsanity.com/paper/2105.01466/) (2021)
- [A Physiologically-Adapted Gold Standard for Arousal during Stress](https://mlsanity.com/paper/2107.12964/) (2021)
- [Insights on Modelling Physiological, Appraisal, and Affective Indicators of Stress using Audio Features](https://mlsanity.com/paper/2205.04328/) (2022)
- [Automatic Emotion Modelling in Written Stories](https://mlsanity.com/paper/2212.11382/) (2022)
- [Improving Personalisation in Valence and Arousal Prediction using Data Augmentation](https://mlsanity.com/paper/2404.09042/) (2024)
- [Exploiting Diverse Feature for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2308.13421/) (2023)
- [From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health](https://mlsanity.com/paper/2609.25186/) (2026)
- [VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale](https://mlsanity.com/paper/2609.32016/) (2026)

## Abstract (arXiv)

Multimodal Sentiment Analysis (MuSe) 2021 is a challenge focusing on the tasks of sentiment and emotion, as well as physiological-emotion and emotion-based stress recognition through more comprehensively integrating the audio-visual, language, and biological signal modalities. The purpose of MuSe 2021 is to bring together communities from different disciplines; mainly, the audio-visual emotion recognition community (signal-based), the sentiment analysis community (symbol-based), and the health informatics community. We present four distinct sub-challenges: MuSe-Wilder and MuSe-Stress which focus on continuous emotion (valence and arousal) prediction; MuSe-Sent, in which participants recognise five classes each for valence and arousal; and MuSe-Physio, in which the novel aspect of `physiological-emotion' is to be predicted. For this years' challenge, we utilise the MuSe-CaR dataset focusing on user-generated reviews and introduce the Ulm-TSST dataset, which displays people in stressful depositions. This paper also provides detail on the state-of-the-art feature sets extracted from these datasets for utilisation by our baseline model, a Long Short-Term Memory-Recurrent Neural Network. For each sub-challenge, a competitive baseline for participants is set; namely, on test, we report a Concordance Correlation Coefficient (CCC) of .4616 CCC for MuSe-Wilder; .4717 CCC for MuSe-Stress, and .4606 CCC for MuSe-Physio. For MuSe-Sent an F1 score of 32.82 % is obtained.