# Bias and Fairness on Multimodal Emotion Detection Algorithms

arXiv: https://arxiv.org/abs/2205.08383 · HTML: https://mlsanity.com/paper/2205.08383/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 11 May 2022.

## Builds on (most cited first, 15 of 15)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias](https://mlsanity.com/paper/1810.01943/) (2018)
- [AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients](https://mlsanity.com/paper/2010.07468/) (2020)
- [Towards Learning a Universal Non-Semantic Representation of Speech](https://mlsanity.com/paper/2002.12764/) (2020)
- [EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa](https://mlsanity.com/paper/2108.12009/) (2021)
- [Multimodal Speech Emotion Recognition Using Audio and Text](https://mlsanity.com/paper/1810.04635/) (2018)
- [Responsible AI: Gender bias assessment in emotion recognition](https://mlsanity.com/paper/2103.11436/) (2021)
- [Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning](https://mlsanity.com/paper/1804.05788/) (2018)
- [Analyzing the Influence of Dataset Composition for Emotion Recognition](https://mlsanity.com/paper/2103.03700/) (2021)

## Built on by (most cited first, 10 of 10)

- [A Survey on Safe Multi-Modal Learning System](https://mlsanity.com/paper/2402.05355/) (2024)
- [Exploring Vision Language Models for Facial Attribute Recognition: Emotion, Race, Gender, and Age](https://mlsanity.com/paper/2410.24148/) (2024)
- [Is It Still Fair? Investigating Gender Fairness in Cross-Corpus Speech Emotion Recognition](https://mlsanity.com/paper/2501.00995/) (2025)
- [Privacy against Real-Time Speech Emotion Detection via Acoustic Adversarial Evasion of Machine Learning](https://mlsanity.com/paper/2211.09273/) (2022)
- [More is Less? A Simulation-Based Approach to Dynamic Interactions between Biases in Multimodal Models](https://mlsanity.com/paper/2412.17505/) (2024)
- [Privacy-Preserving Empathy Detection in Video Interactions](https://mlsanity.com/paper/2504.10808/) (2025)
- [FairSSL: Fair Multimodal Self-Supervised Learning](https://mlsanity.com/paper/2508.16748/) (2025)
- [Affect Models Have Weak Generalizability to Atypical Speech](https://mlsanity.com/paper/2504.16283/) (2025)
- [TrustSkin: A Fairness Pipeline for Trustworthy Facial Affect Analysis Across Skin Tone](https://mlsanity.com/paper/2505.20637/) (2025)
- [A Baseline Multimodal Approach to Emotion Recognition in Conversations](https://mlsanity.com/paper/2602.00914/) (2026)

## Abstract (arXiv)

Numerous studies have shown that machine learning algorithms can latch onto protected attributes such as race and gender and generate predictions that systematically discriminate against one or more groups. To date the majority of bias and fairness research has been on unimodal models. In this work, we explore the biases that exist in emotion recognition systems in relationship to the modalities utilized, and study how multimodal approaches affect system bias and fairness. We consider audio, text, and video modalities, as well as all possible multimodal combinations of those, and find that text alone has the least bias, and accounts for the majority of the models' performances, raising doubts about the worthiness of multimodal emotion recognition systems when bias and fairness are desired alongside model performance.