# When Voice Matters: Evidence of Gender Disparity in Positional Bias of SpeechLLMs

arXiv: https://arxiv.org/abs/2510.02398 · HTML: https://mlsanity.com/paper/2510.02398/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2025.

## Built on by (most cited first, 5 of 5)

- [VoxSafeBench: Not Just What Is Said, but Who, How, and Where](https://mlsanity.com/paper/2604.14548/) (2026)
- [The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMs](https://mlsanity.com/paper/2603.16941/) (2026)
- [Lost in Phonation: Voice Quality Variation as an Evaluation Dimension for Speech Foundation Models](https://mlsanity.com/paper/2510.25577/) (2025)
- [Voice or Stereotype? Disentangling Acoustic and Content-Based Gender in Speech-to-Speech Models](https://mlsanity.com/paper/2609.09263/) (2026)
- [Personalization Without Stereotyping: Do Speech LLMs Know When to Turn a Deaf Ear?](https://mlsanity.com/paper/2610.13550/) (2026)

## Abstract (arXiv)

The rapid development of SpeechLLM-based conversational AI systems has created a need for robustly benchmarking these efforts, including aspects of fairness and bias. At present, such benchmarks typically rely on multiple choice question answering (MCQA). In this paper, we present the first token-level probabilistic evaluation and response-based study of several issues affecting the use of MCQA in SpeechLLM benchmarking: 1) we examine how model temperature and prompt design affect gender and positional bias on an MCQA gender-bias benchmark; 2) we examine how these biases are affected by the gender of the input voice; and 3) we study to what extent observed trends carry over to a second gender-bias benchmark. Our results show that concerns about positional bias from the text domain are equally valid in the speech domain. We also find the effect to be stronger for female voices than for male voices. To our knowledge, this is the first study to isolate positional bias effects in SpeechLLM-based gender-bias benchmarks. We conclude that current MCQA benchmarks do not account for speech-based bias and alternative strategies are needed to ensure fairness towards all users.