mlsanityData to 14 Oct 2026

Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Models

arXiv 2407.06957 ↗First posted 9 Jul 2024eess.AS cs.CL cs.CYcited by 21

Not on the map. Why?

Abstract (arXiv)

Speech Integrated Large Language Models (SILLMs) combine large language models with speech perception to perform diverse tasks, such as emotion recognition to speaker verification, demonstrating universal audio understanding capability. However, these models may amplify biases present in training data, potentially leading to biased access to information for marginalized groups. This work introduces a curated spoken bias evaluation toolkit and corresponding dataset. We evaluate gender bias in SILLMs across four semantic-related tasks: speech-to-text translation (STT), spoken coreference resolution (SCR), spoken sentence continuation (SSC), and spoken question answering (SQA). Our analysis reveals that bias levels are language-dependent and vary with different evaluation methods. Our findings emphasize the necessity of employing multiple approaches to comprehensively assess biases in SILLMs, providing insights for developing fairer SILLM systems.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 36 of its 52 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

36 of 52 references.

References · 36

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  6. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  7. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  8. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  9. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  10. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  12. 2018Gender Bias in Coreference Resolutioncited by 442
24 more
  1. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
  2. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  3. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  4. 2019Evaluating Gender Bias in Machine Translationcited by 249
  5. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  6. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 199
  7. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  8. 2023Gender bias and stereotypes in Large Language Modelscited by 187
  9. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  10. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 153
  11. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  12. 2023Joint Audio and Speech Understandingcited by 100
  13. 2023DeAR: Debiasing Vision-Language Models with Additive Residualscited by 70
  14. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  15. 2021Quantifying Bias in Automatic Speech Recognitioncited by 50
  16. 2021NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender-Neutral Alternativescited by 46
  17. 2019Gender Representation in French Broadcast Corpora and Its Impact on ASR Performancecited by 23
  18. 2023CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Modelscited by 23
  19. 2021Don't speak too fast: The impact of data bias on self-supervised speech modelscited by 20
  20. 2024Emo-bias: A Large Scale Evaluation of Social Bias on Speech Emotion Recognitioncited by 20
  21. 2020Evaluating Gender Bias in Speech Translationcited by 19
  22. 2024On the social bias of speech self-supervised modelscited by 16
  23. 2022A Study of Gender Impact in Self-supervised Models for Speech-to-Text Systemscited by 15
  24. 2024PoliTune: Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in Large Language Modelscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 10.

21 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  2. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  3. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  4. 2026All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluationcited by 8
  5. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  6. 2025Mitigating Subgroup Disparities in Multi-Label Speech Emotion Recognition: A Pseudo-Labeling and Unsupervised Learning Approachcited by 5
  7. 2025SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Modelscited by 5
  8. 2026Causal Tracing of Audio-Text Fusion in Large Audio Language Modelscited by 5
  9. 2025Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variationscited by 4
  10. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  11. 2025EMO-Debias: Benchmarking Gender Debiasing Techniques in Multi-Label Speech Emotion Recognitioncited by 3
  12. 2026VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speechcited by 3
9 more
  1. 2026MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Modelscited by 2
  2. 2025CO-VADA: A Confidence-Oriented Voice Augmentation Debiasing Approach for Fair Speech Emotion Recognitioncited by 1
  3. 2025Acoustic-based Gender Differentiation in Speech-aware Language Modelscited by 1
  4. 2025TAU: A Benchmark for Cultural Sound Understanding Beyond Semanticscited by 1
  5. 2026Greater accessibility can amplify discrimination in generative AIcited by 1
  6. 2025Speak Your Mind: The Speech Continuation Task as a Probe of Voice-Based Model Biascited by 0
  7. 2026Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Modelscited by 0
  8. 2026Measuring Fairness in Large Audio Language Models via Semantic-Aware Bias Estimationcited by 0
  9. 2026Personalization Without Stereotyping: Do Speech LLMs Know When to Turn a Deaf Ear?cited by 0