# Explainable Speech Emotion Recognition: Weighted Attribute Fairness to Model Demographic Contributions to Social Bias

arXiv: https://arxiv.org/abs/2604.19763 · HTML: https://mlsanity.com/paper/2604.19763/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 26 Mar 2026.

## Builds on (most cited first, 20 of 20)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Fairness in Machine Learning: A Survey](https://mlsanity.com/paper/2010.04053/) (2020)
- [A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle](https://mlsanity.com/paper/1901.10002/) (2019)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [Inherent Tradeoffs in Learning Fair Representations](https://mlsanity.com/paper/1906.08386/) (2019)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding](https://mlsanity.com/paper/2111.02735/) (2021)
- [Fairness and Bias in Algorithmic Hiring: a Multidisciplinary Survey](https://mlsanity.com/paper/2309.13933/) (2023)
- [Emo-bias: A Large Scale Evaluation of Social Bias on Speech Emotion Recognition](https://mlsanity.com/paper/2406.05065/) (2024)
- [Speech Emotion Recognition using Supervised Deep Recurrent System for Mental Health Monitoring](https://mlsanity.com/paper/2208.12812/) (2022)
- [EMO-SUPERB: An In-depth Look at Speech Emotion Recognition](https://mlsanity.com/paper/2402.13018/) (2024)
- [Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition](https://mlsanity.com/paper/2310.18877/) (2023)
- [GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations](https://mlsanity.com/paper/2503.20919/) (2025)
- [Large Language Models for Depression Recognition in Spoken Language Integrating Psychological Knowledge](https://mlsanity.com/paper/2505.22863/) (2025)

## Abstract (arXiv)

Speech Emotion Recognition (SER) systems have growing applications in sensitive domains such as mental health and education, where biased predictions can cause harm. Traditional fairness metrics, such as Equalised Odds and Demographic Parity, often overlook the joint dependency between demographic attributes and model predictions. We propose a fairness modelling approach for SER that explicitly captures allocative bias by learning the joint relationship between demographic attributes and model error. We validate our fairness metric on synthetic data, then apply it to evaluate HuBERT and WavLM models finetuned on the CREMA-D dataset. Our results indicate that the proposed fairness model captures more mutual information between protected attributes and biases and quantifies the absolute contribution of individual attributes to bias in SSL-based SER models. Additionally, our analysis reveals indications of gender bias in both HuBERT and WavLM.