# Evaluating Bias in Phoneme-Based Automatic Speech Recognition Systems: An Analysis of IPA Transcription Models

arXiv: https://arxiv.org/abs/2606.11639 · HTML: https://mlsanity.com/paper/2606.11639/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Builds on (most cited first, 22 of 22)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Universal Phone Recognition with a Multilingual Allophone System](https://mlsanity.com/paper/2002.11800/) (2020)
- [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition](https://mlsanity.com/paper/2109.11680/) (2021)
- [Quantifying Bias in Automatic Speech Recognition](https://mlsanity.com/paper/2103.15122/) (2021)
- [The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR](https://mlsanity.com/paper/2303.18110/) (2023)
- [ZIPA: A family of efficient models for multilingual phone recognition](https://mlsanity.com/paper/2505.23170/) (2025)
- [MediaSpeech: Multilanguage ASR Benchmark and Dataset](https://mlsanity.com/paper/2103.16193/) (2021)
- [The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any language](https://mlsanity.com/paper/2311.08323/) (2023)
- [Anatomy of Industrial Scale Multilingual ASR](https://mlsanity.com/paper/2404.09841/) (2024)
- [Exploring data augmentation in bias mitigation against non-native-accented speech](https://mlsanity.com/paper/2312.15499/) (2023)
- [WAXAL: A Large-Scale Multilingual African Language Speech Corpus](https://mlsanity.com/paper/2602.02734/) (2026)
- [AlloVera: A Multilingual Allophone Database](https://mlsanity.com/paper/2004.08031/) (2020)
- [Dolphin: A Large-Scale Automatic Speech Recognition Model for Eastern Languages](https://mlsanity.com/paper/2503.20212/) (2025)
- [Adapting Whisper for Regional Dialects: Enhancing Public Services for Vulnerable Populations in the United Kingdom](https://mlsanity.com/paper/2501.08502/) (2025)
- [Sonos Voice Control Bias Assessment Dataset: A Methodology for Demographic Bias Assessment in Voice Assistants](https://mlsanity.com/paper/2405.19342/) (2024)
- [IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modeling](https://mlsanity.com/paper/2504.03036/) (2025)
- [Automatic Speech Recognition Biases in Newcastle English: an Error Analysis](https://mlsanity.com/paper/2506.16558/) (2025)
- [Toward Responsible ASR for African American English Speakers: A Scoping Review of Bias and Equity in Speech Technology](https://mlsanity.com/paper/2508.18288/) (2025)
- [Advanced Modeling of Interlanguage Speech Intelligibility Benefit with L1-L2 Multi-Task Learning Using Differentiable K-Means for Accent-Robust Discrete Token-Based ASR](https://mlsanity.com/paper/2601.19767/) (2026)
- [Linguistically Informed Tokenization Improves ASR for Underresourced Languages](https://mlsanity.com/paper/2510.06461/) (2025)
- [A Sociophonetic Analysis of Racial Bias in Commercial ASR Systems Using the Pacific Northwest English Corpus](https://mlsanity.com/paper/2510.22495/) (2025)

## Abstract (arXiv)

As automatic speech recognition (ASR) systems shift toward multilingual support and low-resource language modeling, phoneme-based layers serve as a critical language-agnostic foundation. However, most evaluations of ASR's demographic biases related to race, age, gender, and accent focus on standard grapheme-based ASR systems with comparatively little emphasis on phoneme-based systems. In this study, we evaluate the performance of WhisperIPA and ZIPA, two state-of-the-art open-source systems that generate International Phonetic Alphabet (IPA) transcriptions. Our evaluation includes existing multilingual speech corpora and demographically annotated English-language corpora, comparing model-generated IPA transcriptions against grapheme-to-phoneme (G2P) systems using both standard phoneme error rate (PER) and a proposed Soft PER metric that tolerates linguistically similar phoneme substitutions. Our analysis examines how performance varies across language, gender, accent, ethnicity, and age, revealing persistent disparities even after accounting for acceptable phonemic variation. These findings, while limited, provide insight into potential sources of bias and inform the development of more inclusive and linguistically robust phoneme-based ASR systems. Our code and data are publicly available.