# Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition

arXiv: https://arxiv.org/abs/2604.21276 · HTML: https://mlsanity.com/paper/2604.21276/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 23 Apr 2026.

## Builds on (most cited first, 13 of 13)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Careless Whisper: Speech-to-Text Hallucination Harms](https://mlsanity.com/paper/2402.08021/) (2024)
- [Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Models](https://mlsanity.com/paper/2401.01572/) (2024)
- [Granite-speech: open-source speech-aware LLMs with strong English ASR capabilities](https://mlsanity.com/paper/2505.08699/) (2025)
- [Speech Robust Bench: A Robustness Benchmark For Speech Recognition](https://mlsanity.com/paper/2403.07937/) (2024)
- [Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models](https://mlsanity.com/paper/2502.12414/) (2025)
- [Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward](https://mlsanity.com/paper/2411.03866/) (2024)
- [Svarah: Evaluating English ASR Systems on Indian Accents](https://mlsanity.com/paper/2305.15760/) (2023)
- [Hallucination Benchmark for Speech Foundation Models](https://mlsanity.com/paper/2510.16567/) (2025)
- [Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations](https://mlsanity.com/paper/2602.01030/) (2026)

## Built on by (most cited first, 4 of 4)

- [Temporal Taxation Compounds Under Post-Training Compression of Whisper Models](https://mlsanity.com/paper/2609.28739/) (2026)
- [Pruning for Efficiency, Paying in Fairness: Demographic Disparities in Pruned Speech-LLMs](https://mlsanity.com/paper/2609.38106/) (2026)
- [Fairness Beyond a Single Run: Training-Seed Variability in Speech LLM Adaptation](https://mlsanity.com/paper/2609.38976/) (2026)
- [Personalization Without Stereotyping: Do Speech LLMs Know When to Turn a Deaf Ear?](https://mlsanity.com/paper/2610.13550/) (2026)

## Abstract (arXiv)

As pretrained large language models replace task-specific decoders in speech recognition, a critical question arises: do their text-derived priors make recognition fairer or more biased across demographic groups? We evaluate nine models spanning three architectural generations (CTC with no language model, encoder-decoder with an implicit LM, and LLM-based with an explicit pretrained decoder) on about 43,000 utterances across five demographic axes (ethnicity, accent, gender, age, first language) using Common Voice 24 and Meta's Fair-Speech, a controlled-prompt dataset that eliminates vocabulary confounds. On clean audio, three findings challenge assumptions: LLM decoders do not amplify racial bias (Granite-8B has the best ethnicity fairness, max/min WER = 2.28); Whisper exhibits pathological hallucination on Indian-accented speech with a non-monotonic insertion-rate spike to 9.62% at large-v3; and audio compression predicts accent fairness more than LLM scale. We then stress-test these findings under 12 acoustic degradation conditions (noise, reverberation, silence injection, chunk masking) across both datasets, totaling 216 inference runs. Severe degradation paradoxically compresses fairness gaps as all groups converge to high WER, but silence injection amplifies Whisper's accent bias up to 4.64x by triggering demographic-selective hallucination. Under masking, Whisper enters catastrophic repetition loops (86% of 51,797 insertions) while explicit-LLM decoders produce 38x fewer insertions with near-zero repetition; high-compression audio encoding (Q-former) reintroduces repetition pathology even in LLM decoders. These results suggest that audio encoder design, not LLM scaling, is the primary lever for equitable and robust speech recognition.