# The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation

arXiv: https://arxiv.org/abs/2305.03369 · HTML: https://mlsanity.com/paper/2305.03369/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 71 references that are in the map.
- First posted on arXiv 5 May 2023.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [Speech Emotion Recognition using Self-Supervised Features](https://mlsanity.com/paper/2202.03896/) (2022)
- [PySBD: Pragmatic Sentence Boundary Disambiguation](https://mlsanity.com/paper/2010.09657/) (2020)
- [The INTERSPEECH 2021 Computational Paralinguistics Challenge: COVID-19 Cough, COVID-19 Speech, Escalation & Primates](https://mlsanity.com/paper/2102.13468/) (2021)
- [The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress](https://mlsanity.com/paper/2104.07123/) (2021)
- [The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress](https://mlsanity.com/paper/2207.05691/) (2022)
- [EmoNet: A Transfer Learning Framework for Multi-Corpus Speech Emotion Recognition](https://mlsanity.com/paper/2103.08310/) (2021)
- [DeepSpectrumLite: A Power-Efficient Transfer Learning Framework for Embedded Speech and Audio Processing from Decentralised Data](https://mlsanity.com/paper/2104.11629/) (2021)
- [Hybrid Multimodal Feature Extraction, Mining and Fusion for Sentiment Analysis](https://mlsanity.com/paper/2208.03051/) (2022)
- ["So You Think You're Funny?": Rating the Humour Quotient in Standup Comedy](https://mlsanity.com/paper/2110.12765/) (2021)
- [Hybrid Multimodal Fusion for Humor Detection](https://mlsanity.com/paper/2209.11949/) (2022)
- [MuSe-Toolbox: The Multimodal Sentiment Analysis Continuous Annotation Fusion and Discrete Class Transformation Toolbox](https://mlsanity.com/paper/2107.11757/) (2021)
- [Beyond One-Size-Fits-All: A Survey of Personalized Affective Computing in Human-Agent Interaction](https://mlsanity.com/paper/2304.00377/) (2023)

## Built on by (most cited first, 12 of 12)

- [The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition](https://mlsanity.com/paper/2402.19344/) (2024)
- [Can Large Multimodal Models Uncover Deep Semantics Behind Images?](https://mlsanity.com/paper/2402.11281/) (2024)
- [Unimodal Multi-Task Fusion for Emotional Mimicry Intensity Prediction](https://mlsanity.com/paper/2403.11879/) (2024)
- [Semantic Matters: Multimodal Features for Affective Analysis](https://mlsanity.com/paper/2504.11460/) (2025)
- [Modeling Emotional Trajectories in Written Stories Utilizing Transformers and Weakly-Supervised Learning](https://mlsanity.com/paper/2406.02251/) (2024)
- [The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognition](https://mlsanity.com/paper/2406.07753/) (2024)
- [Improving Personalisation in Valence and Arousal Prediction using Data Augmentation](https://mlsanity.com/paper/2404.09042/) (2024)
- [The 2026 ACII Dyadic Conversations (DaiKon) Workshop & Challenge](https://mlsanity.com/paper/2605.02672/) (2026)
- [Exploiting Diverse Feature for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2308.13421/) (2023)
- [From Generalized Laughter to Personalized Chuckles: Unleashing the Power of Data Fusion in Subjective Humor Detection](https://mlsanity.com/paper/2312.11296/) (2023)
- [This Paper Had the Smartest Reviewers -- Flattery Detection Utilising an Audio-Textual Transformer-Based Approach](https://mlsanity.com/paper/2406.17667/) (2024)
- [Evaluating multimodal emotion recognition in proactive conversational agents: A user study](https://mlsanity.com/paper/2605.20200/) (2026)

## Abstract (arXiv)

The MuSe 2023 is a set of shared tasks addressing three different contemporary multimodal affect and sentiment analysis problems: In the Mimicked Emotions Sub-Challenge (MuSe-Mimic), participants predict three continuous emotion targets. This sub-challenge utilises the Hume-Vidmimic dataset comprising of user-generated videos. For the Cross-Cultural Humour Detection Sub-Challenge (MuSe-Humour), an extension of the Passau Spontaneous Football Coach Humour (Passau-SFCH) dataset is provided. Participants predict the presence of spontaneous humour in a cross-cultural setting. The Personalisation Sub-Challenge (MuSe-Personalisation) is based on the Ulm-Trier Social Stress Test (Ulm-TSST) dataset, featuring recordings of subjects in a stressed situation. Here, arousal and valence signals are to be predicted, whereas parts of the test labels are made available in order to facilitate personalisation. MuSe 2023 seeks to bring together a broad audience from different research communities such as audio-visual emotion recognition, natural language processing, signal processing, and health informatics. In this baseline paper, we introduce the datasets, sub-challenges, and provided feature sets. As a competitive baseline system, a Gated Recurrent Unit (GRU)-Recurrent Neural Network (RNN) is employed. On the respective sub-challenges' test datasets, it achieves a mean (across three continuous intensity targets) Pearson's Correlation Coefficient of .4727 for MuSe-Mimic, an Area Under the Curve (AUC) value of .8310 for MuSe-Humor and Concordance Correlation Coefficient (CCC) values of .7482 for arousal and .7827 for valence in the MuSe-Personalisation sub-challenge.