# MuSe-Toolbox: The Multimodal Sentiment Analysis Continuous Annotation Fusion and Discrete Class Transformation Toolbox

arXiv: https://arxiv.org/abs/2107.11757 · HTML: https://mlsanity.com/paper/2107.11757/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 57 references that are in the map.
- First posted on arXiv 25 Jul 2021.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild](https://mlsanity.com/paper/1901.02839/) (2019)
- [The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress](https://mlsanity.com/paper/2104.07123/) (2021)
- [The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and Improvements](https://mlsanity.com/paper/2101.06053/) (2021)
- [Nonsmooth Analysis and Subgradient Methods for Averaging in Dynamic Time Warping Spaces](https://mlsanity.com/paper/1701.06393/) (2017)
- [An Estimation of Online Video User Engagement from Features of Continuous Emotions](https://mlsanity.com/paper/2105.01633/) (2021)

## Built on by (most cited first, 8 of 8)

- [The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress](https://mlsanity.com/paper/2207.05691/) (2022)
- [Continuous Emotion Recognition using Visual-audio-linguistic information: A Technical Report for ABAW3](https://mlsanity.com/paper/2203.13031/) (2022)
- [The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation](https://mlsanity.com/paper/2305.03369/) (2023)
- [Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion](https://mlsanity.com/paper/2307.13463/) (2023)
- [Towards Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results](https://mlsanity.com/paper/2209.14272/) (2022)
- [Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognition](https://mlsanity.com/paper/2311.10119/) (2023)
- [A Physiologically-Adapted Gold Standard for Arousal during Stress](https://mlsanity.com/paper/2107.12964/) (2021)
- [Can Valence Reflect Morality in Natural Language? A Preliminary Annotation Study](https://mlsanity.com/paper/2607.20461/) (2026)

## Abstract (arXiv)

We introduce the MuSe-Toolbox - a Python-based open-source toolkit for creating a variety of continuous and discrete emotion gold standards. In a single framework, we unify a wide range of fusion methods and propose the novel Rater Aligned Annotation Weighting (RAAW), which aligns the annotations in a translation-invariant way before weighting and fusing them based on the inter-rater agreements between the annotations. Furthermore, discrete categories tend to be easier for humans to interpret than continuous signals. With this in mind, the MuSe-Toolbox provides the functionality to run exhaustive searches for meaningful class clusters in the continuous gold standards. To our knowledge, this is the first toolkit that provides a wide selection of state-of-the-art emotional gold standard methods and their transformation to discrete classes. Experimental results indicate that MuSe-Toolbox can provide promising and novel class formations which can be better predicted than hard-coded classes boundaries with minimal human intervention. The implementation (1) is out-of-the-box available with all dependencies using a Docker container (2).