# VoiceBBQ: Investigating Effect of Content and Acoustics in Social Bias of Spoken Language Model

arXiv: https://arxiv.org/abs/2509.21108 · HTML: https://mlsanity.com/paper/2509.21108/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 26 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 10 of 10)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://mlsanity.com/paper/2306.07691/) (2023)
- [Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias](https://mlsanity.com/paper/2308.00225/) (2023)
- [Evaluating Gender Bias in Speech Translation](https://mlsanity.com/paper/2010.14465/) (2020)
- [Speech-based Age and Gender Prediction with Transformers](https://mlsanity.com/paper/2306.16962/) (2023)
- [FairSSD: Understanding Bias in Synthetic Speech Detectors](https://mlsanity.com/paper/2404.10989/) (2024)

## Built on by (most cited first, 6 of 6)

- [Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation](https://mlsanity.com/paper/2511.21517/) (2025)
- [VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech](https://mlsanity.com/paper/2604.17248/) (2026)
- [Greater accessibility can amplify discrimination in generative AI](https://mlsanity.com/paper/2603.22260/) (2026)
- [Voice or Stereotype? Disentangling Acoustic and Content-Based Gender in Speech-to-Speech Models](https://mlsanity.com/paper/2609.09263/) (2026)
- [HEAR: Real Voices, Real Bias: A Large-Scale Human-Recorded, Demographically Diverse Benchmark for Audio Language Models](https://mlsanity.com/paper/2609.35952/) (2026)
- [Personalization Without Stereotyping: Do Speech LLMs Know When to Turn a Deaf Ear?](https://mlsanity.com/paper/2610.13550/) (2026)

## Abstract (arXiv)

We introduce VoiceBBQ, a spoken extension of the BBQ (Bias Benchmark for Question Answering) - a dataset that measures social bias by presenting ambiguous or disambiguated contexts followed by questions that may elicit stereotypical responses. Due to the nature of speech, social bias in Spoken Language Models (SLMs) can emerge from two distinct sources: 1) content aspect and 2) acoustic aspect. The dataset converts every BBQ context into controlled voice conditions, enabling per-axis accuracy, bias, and consistency scores that remain comparable to the original text benchmark. Using VoiceBBQ, we evaluate two SLMs - LLaMA-Omni and Qwen2-Audio - and observe architectural contrasts: LLaMA-Omni resists acoustic bias while amplifying gender and accent bias, whereas Qwen2-Audio substantially dampens these cues while preserving content fidelity. VoiceBBQ thus provides a compact, drop-in testbed for jointly diagnosing content and acoustic bias across spoken language models.