# Improving Fairness and Robustness in End-to-End Speech Recognition through unsupervised clustering

arXiv: https://arxiv.org/abs/2306.06083 · HTML: https://mlsanity.com/paper/2306.06083/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 22 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 8 of 8)

- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Quantifying Bias in Automatic Speech Recognition](https://mlsanity.com/paper/2103.15122/) (2021)
- [Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Model](https://mlsanity.com/paper/1712.01541/) (2017)
- [Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparities](https://mlsanity.com/paper/2207.11345/) (2022)
- [Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptions](https://mlsanity.com/paper/2111.09983/) (2021)
- [ASR4REAL: An extended benchmark for speech models](https://mlsanity.com/paper/2110.08583/) (2021)

## Built on by (most cited first, 11 of 11)

- [Towards measuring fairness in speech recognition: Fair-Speech dataset](https://mlsanity.com/paper/2408.12734/) (2024)
- [Speech Robust Bench: A Robustness Benchmark For Speech Recognition](https://mlsanity.com/paper/2403.07937/) (2024)
- [A Contrastive Learning Approach to Mitigate Bias in Speech Models](https://mlsanity.com/paper/2406.14686/) (2024)
- [Explaining Speech Classification Models via Word-Level Audio Segments and Paralinguistic Features](https://mlsanity.com/paper/2309.07733/) (2023)
- [FairJob: A Real-World Dataset for Fairness in Online Systems](https://mlsanity.com/paper/2407.03059/) (2024)
- [Fairness of Automatic Speech Recognition: Looking Through a Philosophical Lens](https://mlsanity.com/paper/2508.07143/) (2025)
- [Using Backbone Foundation Model for Evaluating Fairness in Chest Radiography Without Demographic Data](https://mlsanity.com/paper/2408.16130/) (2024)
- [CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognition](https://mlsanity.com/paper/2502.01777/) (2025)
- [CO-VADA: A Confidence-Oriented Voice Augmentation Debiasing Approach for Fair Speech Emotion Recognition](https://mlsanity.com/paper/2506.06071/) (2025)
- [Position: Fairness Failure in Generative Models is an Evaluation Problem](https://mlsanity.com/paper/2608.16974/) (2026)
- [When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Models](https://mlsanity.com/paper/2609.27382/) (2026)

## Abstract (arXiv)

The challenge of fairness arises when Automatic Speech Recognition (ASR) systems do not perform equally well for all sub-groups of the population. In the past few years there have been many improvements in overall speech recognition quality, but without any particular focus on advancing Equality and Equity for all user groups for whom systems do not perform well. ASR fairness is therefore also a robustness issue. Meanwhile, data privacy also takes priority in production systems. In this paper, we present a privacy preserving approach to improve fairness and robustness of end-to-end ASR without using metadata, zip codes, or even speaker or utterance embeddings directly in training. We extract utterance level embeddings using a speaker ID model trained on a public dataset, which we then use in an unsupervised fashion to create acoustic clusters. We use cluster IDs instead of speaker utterance embeddings as extra features during model training, which shows improvements for all demographic groups and in particular for different accents.