mlsanityData to 14 Oct 2026

The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMs

arXiv 2603.16941 ↗First posted 15 Mar 2026eess.AS cs.CL cs.SDcited by 2

Not on the map. Why?

Abstract (arXiv)

Speech Large Language Models (SpeechLLMs) process spoken input directly, retaining cues such as accent and perceived gender that were previously removed in cascaded pipelines. This introduces speaker identity dependent variation in responses. We present a large-scale intersectional evaluation of accent and gender bias in three SpeechLLMs using 2,880 controlled interactions across six English accents and two gender presentations, keeping linguistic content constant through voice cloning. Using pointwise LLM-judge ratings, pairwise comparisons, and Best-Worst Scaling with human validation, we detect recurring directional disparities. Eastern European-accented speech receives lower helpfulness scores, particularly for female-presenting voices. Responses remain polite but differ in helpfulness. While LLM judges capture the directional trend of these biases, human evaluators exhibit significantly higher sensitivity, showing stronger accent-level contrasts.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 15 Mar 2026.

Builds on

23 of 34 references.

References · 23

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2025Qwen3-Omni Technical Reportcited by 497
  5. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  6. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  7. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  8. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  9. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  10. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  11. 2017Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotationcited by 79
  12. 2024A Survey on Speech Large Language Models for Understandingcited by 71
11 more
  1. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
  2. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  3. 2024Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluationcited by 41
  4. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 41
  5. 2025LFM2 Technical Reportcited by 37
  6. 2023The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASRcited by 36
  7. 2024Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Modelscited by 16
  8. 2025Do Bias Benchmarks Generalise? Evidence from Voice-based Evaluation of Gender Bias in SpeechLLMscited by 7
  9. 2023Concept-Guided Chain-of-Thought Prompting for Pairwise Comparison Scoring of Texts with Large Language Modelscited by 5
  10. 2025When Voice Matters: Evidence of Gender Disparity in Positional Bias of SpeechLLMscited by 5
  11. 2026Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variationscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Auditing Bias and Safety in Voice AI Customer Carecited by 0
  2. 2026Almost Human, Except When It Matters: VoxParity and the Decisions a Voice Should Changecited by 0