# Sonos Voice Control Bias Assessment Dataset: A Methodology for Demographic Bias Assessment in Voice Assistants

arXiv: https://arxiv.org/abs/2405.19342 · HTML: https://mlsanity.com/paper/2405.19342/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 59 references that are in the map.
- First posted on arXiv 14 May 2024.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [Quantifying Bias in Automatic Speech Recognition](https://mlsanity.com/paper/2103.15122/) (2021)
- [Spoken Language Understanding on the Edge](https://mlsanity.com/paper/1810.12735/) (2018)
- [Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance](https://mlsanity.com/paper/1908.08717/) (2019)
- [Improving Readability for Automatic Speech Recognition Transcription](https://mlsanity.com/paper/2004.04438/) (2020)
- [Hey ASR System! Why Aren't You More Inclusive? Automatic Speech Recognition Systems' Bias and Proposed Bias Mitigation Techniques. A Literature Review](https://mlsanity.com/paper/2211.09511/) (2022)
- [Model-Based Approach for Measuring the Fairness in ASR](https://mlsanity.com/paper/2109.09061/) (2021)
- [See what I'm saying? Comparing Intelligent Personal Assistant use for Native and Non-Native Language Speakers](https://mlsanity.com/paper/2006.06328/) (2020)
- [NISP: A Multi-lingual Multi-accent Dataset for Speaker Profiling](https://mlsanity.com/paper/2007.06021/) (2020)
- [Deaf, Hard of Hearing, and Hearing Perspectives on using Automatic Speech Recognition in Conversation](https://mlsanity.com/paper/1909.01176/) (2019)
- [Tensor decomposition for minimization of E2E SLU model toward on-device processing](https://mlsanity.com/paper/2306.01247/) (2023)

## Built on by (most cited first, 6 of 6)

- [Evaluating Bias in Spoken Dialogue LLMs for Real-World Decisions and Recommendations](https://mlsanity.com/paper/2510.02352/) (2025)
- [Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models](https://mlsanity.com/paper/2604.22631/) (2026)
- [Where Do Self-Supervised Speech Models Become Unfair?](https://mlsanity.com/paper/2604.18249/) (2026)
- [Responsible Benchmarking of Fairness for Automatic Speech Recognition](https://mlsanity.com/paper/2605.10615/) (2026)
- [Evaluating Bias in Phoneme-Based Automatic Speech Recognition Systems: An Analysis of IPA Transcription Models](https://mlsanity.com/paper/2606.11639/) (2026)
- [ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge](https://mlsanity.com/paper/2606.24648/) (2026)

## Abstract (arXiv)

Recent works demonstrate that voice assistants do not perform equally well for everyone, but research on demographic robustness of speech technologies is still scarce. This is mainly due to the rarity of large datasets with controlled demographic tags. This paper introduces the Sonos Voice Control Bias Assessment Dataset, an open dataset composed of voice assistant requests for North American English in the music domain (1,038 speakers, 166 hours, 170k audio samples, with 9,040 unique labelled transcripts) with a controlled demographic diversity (gender, age, dialectal region and ethnicity). We also release a statistical demographic bias assessment methodology, at the univariate and multivariate levels, tailored to this specific use case and leveraging spoken language understanding metrics rather than transcription accuracy, which we believe is a better proxy for user experience. To demonstrate the capabilities of this dataset and statistical method to detect demographic bias, we consider a pair of state-of-the-art Automatic Speech Recognition and Spoken Language Understanding models. Results show statistically significant differences in performance across age, dialectal region and ethnicity. Multivariate tests are crucial to shed light on mixed effects between dialectal region, gender and age.