# Implicit Design Choices and Their Impact on Emotion Recognition Model Development and Evaluation

arXiv: https://arxiv.org/abs/2309.03238 · HTML: https://mlsanity.com/paper/2309.03238/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 242 references that are in the map.
- First posted on arXiv 6 Sep 2023.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Differential Privacy Has Disparate Impact on Model Accuracy](https://mlsanity.com/paper/1905.12101/) (2019)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Adversarial Removal of Demographic Attributes from Text Data](https://mlsanity.com/paper/1808.06640/) (2018)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [The Measure and Mismeasure of Fairness](https://mlsanity.com/paper/1808.00023/) (2018)
- [EmotionLines: An Emotion Corpus of Multi-Party Conversations](https://mlsanity.com/paper/1802.08379/) (2018)
- [AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models](https://mlsanity.com/paper/1909.09251/) (2019)
- [Privacy-preserving Neural Representations of Text](https://mlsanity.com/paper/1808.09408/) (2018)
- [Quantifying Interpretability and Trust in Machine Learning Systems](https://mlsanity.com/paper/1901.08558/) (2019)
- [Practical Hidden Voice Attacks against Speech and Speaker Recognition Systems](https://mlsanity.com/paper/1904.05734/) (2019)
- [On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study](https://mlsanity.com/paper/2106.00872/) (2021)
- [Crafting Adversarial Examples For Speech Paralinguistics Applications](https://mlsanity.com/paper/1711.03280/) (2017)
- [Improving Cross-Corpus Speech Emotion Recognition with Adversarial Discriminative Domain Generalization (ADDoG)](https://mlsanity.com/paper/1903.12094/) (2019)
- [Debiasing Methods in Natural Language Understanding Make Bias More Accessible](https://mlsanity.com/paper/2109.04095/) (2021)
- [Adversarial Training for Satire Detection: Controlling for Confounding Variables](https://mlsanity.com/paper/1902.11145/) (2019)
- [MuSE-ing on the Impact of Utterance Ordering On Crowdsourced Emotion Annotations](https://mlsanity.com/paper/1903.11672/) (2019)
- [ConStance: Modeling Annotation Contexts to Improve Stance Classification](https://mlsanity.com/paper/1708.06309/) (2017)
- [Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport](https://mlsanity.com/paper/1812.04597/) (2018)
- [Controlling for Confounders in Multimodal Emotion Classification via Adversarial Learning](https://mlsanity.com/paper/1908.08979/) (2019)
- [The PRIORI Emotion Dataset: Linking Mood to Emotion Detected In-the-Wild](https://mlsanity.com/paper/1806.10658/) (2018)
- [Learning Spontaneity to Improve Emotion Recognition In Speech](https://mlsanity.com/paper/1712.04753/) (2017)
- [Mirroring to Build Trust in Digital Assistants](https://mlsanity.com/paper/1904.01664/) (2019)

## Abstract (arXiv)

Emotion recognition is a complex task due to the inherent subjectivity in both the perception and production of emotions. The subjectivity of emotions poses significant challenges in developing accurate and robust computational models. This thesis examines critical facets of emotion recognition, beginning with the collection of diverse datasets that account for psychological factors in emotion production. To handle the challenge of non-representative training data, this work collects the Multimodal Stressed Emotion dataset, which introduces controlled stressors during data collection to better represent real-world influences on emotion production. To address issues with label subjectivity, this research comprehensively analyzes how data augmentation techniques and annotation schemes impact emotion perception and annotator labels. It further handles natural confounding variables and variations by employing adversarial networks to isolate key factors like stress from learned emotion representations during model training. For tackling concerns about leakage of sensitive demographic variables, this work leverages adversarial learning to strip sensitive demographic information from multimodal encodings. Additionally, it proposes optimized sociological evaluation metrics aligned with cost-effective, real-world needs for model testing. This research advances robust, practical emotion recognition through multifaceted studies of challenges in datasets, labels, modeling, demographic and membership variable encoding in representations, and evaluation. The groundwork has been laid for cost-effective, generalizable emotion recognition models that are less likely to encode sensitive demographic information.