# VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech

arXiv: https://arxiv.org/abs/2604.17248 · HTML: https://mlsanity.com/paper/2604.17248/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 47 references that are in the map.
- First posted on arXiv 19 Apr 2026.

## Builds on (most cited first, 30 of 35)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models](https://mlsanity.com/paper/2507.08128/) (2025)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Step-Audio 2 Technical Report](https://mlsanity.com/paper/2507.16632/) (2025)
- [Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks](https://mlsanity.com/paper/2411.05361/) (2024)
- [Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images](https://mlsanity.com/paper/2402.05779/) (2024)
- [DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment](https://mlsanity.com/paper/2507.02768/) (2025)
- [Quantifying Social Biases Using Templates is Unreliable](https://mlsanity.com/paper/2210.04337/) (2022)
- [Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2505.15957/) (2025)
- [textttModSCAN: Measuring Stereotypical Bias in Large Vision-Language Models from Vision and Language Modalities](https://mlsanity.com/paper/2410.06967/) (2024)
- [Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Models](https://mlsanity.com/paper/2407.06957/) (2024)
- [Emo-bias: A Large Scale Evaluation of Social Bias on Speech Emotion Recognition](https://mlsanity.com/paper/2406.05065/) (2024)
- [Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models](https://mlsanity.com/paper/2408.07665/) (2024)
- [AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models](https://mlsanity.com/paper/2505.16211/) (2025)
- [AHELM: A Holistic Evaluation of Audio-Language Models](https://mlsanity.com/paper/2508.21376/) (2025)
- [Silenced Biases: The Dark Side LLMs Learned to Refuse](https://mlsanity.com/paper/2511.03369/) (2025)
- [VoiceBBQ: Investigating Effect of Content and Acoustics in Social Bias of Spoken Language Model](https://mlsanity.com/paper/2509.21108/) (2025)
- [Mitigating Subgroup Disparities in Multi-Label Speech Emotion Recognition: A Pseudo-Labeling and Unsupervised Learning Approach](https://mlsanity.com/paper/2505.14449/) (2025)
- [Unveiling Biases while Embracing Sustainability: Assessing the Dual Challenges of Automatic Speech Recognition Systems](https://mlsanity.com/paper/2503.00907/) (2025)
- [Debiased Automatic Speech Recognition for Dysarthric Speech via Sample Reweighting with Sample Affinity Test](https://mlsanity.com/paper/2305.13108/) (2023)

## Built on by (most cited first, 3 of 3)

- [Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models](https://mlsanity.com/paper/2604.25591/) (2026)
- [RedVox: Safety and Fairness Gaps in Speech Models Across Languages](https://mlsanity.com/paper/2606.26968/) (2026)
- [Auditing Bias and Safety in Voice AI Customer Care](https://mlsanity.com/paper/2609.04206/) (2026)

## Abstract (arXiv)

Large Audio-Language Models (LALMs) are increasingly integrated into daily applications, yet their generative biases remain underexplored. Existing speech fairness benchmarks rely on synthetic speech and Multiple-Choice Questions (MCQs), both offering a fragmented view of fairness. We propose VIBE, a framework that evaluates generative bias through open-ended tasks such as personalized recommendations, using human-recorded speech. Unlike MCQs, our method allows stereotypical associations to manifest organically without predefined options, making it easily extensible to new tasks. Evaluating 12 state-of-the-art LALMs reveals systematic biases in realistic scenarios. Both gender and accent cues trigger statistically significant distributional shifts, and bias magnitude is strongly task-dependent.