mlsanityData to 7 Oct 2026

Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variations

arXiv 2510.16893 ↗First posted 19 Oct 2025cs.SD cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Large audio-language models (LALMs) extend text-based LLMs with auditory understanding, offering new opportunities for multimodal applications. While their perception, reasoning, and task performance have been widely studied, their safety alignment under paralinguistic variation remains underexplored. This work systematically investigates the role of speaker emotion. We construct a dataset of malicious speech instructions expressed across multiple emotions and intensities, and evaluate several state-of-the-art LALMs. Our results reveal substantial safety inconsistencies: different emotions elicit varying levels of unsafe responses, and the effect of intensity is non-monotonic, with medium expressions often posing the greatest risk. These findings highlight an overlooked vulnerability in LALMs and call for alignment strategies explicitly designed to ensure robustness under emotional variation, a prerequisite for trustworthy deployment in real-world settings.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 37 of its 39 references that are in the map. First posted on arXiv 19 Oct 2025.

Builds on

37 of 39 references.

References · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  8. 2025Qwen2.5-Omni Technical Reportcited by 704
  9. 2024Qwen2-Audio Technical Reportcited by 543
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  12. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
25 more
  1. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  2. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  3. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  4. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  5. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
  6. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  7. 2024Best-of-N Jailbreakingcited by 71
  8. 2024Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Taskscited by 70
  9. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  10. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 43
  11. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  12. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  13. 2024Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attackscited by 35
  14. 2024Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Modelscited by 31
  15. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  16. 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
  17. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
  18. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 25
  19. 2024Building a Taiwanese Mandarin Spoken Language Model: A First Attemptcited by 22
  20. 2024Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Modelscited by 20
  21. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  22. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  23. 2025A Preliminary Exploration with GPT-4o Voice Modecited by 16
  24. 2025JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Modelscited by 11
  25. 2024Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook
  2. 2024Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insights
  3. 2026SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech
  4. 2025"I am bad": Interpreting Stealthy, Universal and Robust Audio Jailbreaks in Audio-Language Models
  5. 2026Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs
  6. 2025Synthetic Voices, Real Threats: Evaluating Large Text-to-Speech Models in Generating Harmful Audio
  7. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts
  8. 2025Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guard