# MuSE-ing on the Impact of Utterance Ordering On Crowdsourced Emotion Annotations

arXiv: https://arxiv.org/abs/1903.11672 · HTML: https://mlsanity.com/paper/1903.11672/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2019.

## Built on by (most cited first, 9 of 9)

- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [MUSER: MUltimodal Stress Detection using Emotion Recognition as an Auxiliary Task](https://mlsanity.com/paper/2105.08146/) (2021)
- [Personalized Prediction of Recurrent Stress Events Using Self-Supervised Learning on Multimodal Time-Series Data](https://mlsanity.com/paper/2307.03337/) (2023)
- [More Similar than Dissimilar: Modeling Annotators for Cross-Corpus Speech Emotion Recognition](https://mlsanity.com/paper/2509.12295/) (2025)
- [What You Feel Is Not What They See: On Predicting Self-Reported Emotion from Third-Party Observer Labels](https://mlsanity.com/paper/2601.21130/) (2026)
- [Best Practices for Noise-Based Augmentation to Improve the Performance of Deployable Speech-Based Emotion Recognition Systems](https://mlsanity.com/paper/2104.08806/) (2021)
- [Implicit Design Choices and Their Impact on Emotion Recognition Model Development and Evaluation](https://mlsanity.com/paper/2309.03238/) (2023)
- [SEER: The Span-based Emotion Evidence Retrieval Benchmark](https://mlsanity.com/paper/2510.03490/) (2025)

## Abstract (arXiv)

Emotion recognition algorithms rely on data annotated with high quality labels. However, emotion expression and perception are inherently subjective. There is generally not a single annotation that can be unambiguously declared "correct". As a result, annotations are colored by the manner in which they were collected. In this paper, we conduct crowdsourcing experiments to investigate this impact on both the annotations themselves and on the performance of these algorithms. We focus on one critical question: the effect of context. We present a new emotion dataset, Multimodal Stressed Emotion (MuSE), and annotate the dataset using two conditions: randomized, in which annotators are presented with clips in random order, and contextualized, in which annotators are presented with clips in order. We find that contextual labeling schemes result in annotations that are more similar to a speaker's own self-reported labels and that labels generated from randomized schemes are most easily predictable by automated systems.