Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variations
Not on the map. Why?
Abstract (arXiv)
Large audio-language models (LALMs) extend text-based LLMs with auditory understanding, offering new opportunities for multimodal applications. While their perception, reasoning, and task performance have been widely studied, their safety alignment under paralinguistic variation remains underexplored. This work systematically investigates the role of speaker emotion. We construct a dataset of malicious speech instructions expressed across multiple emotions and intensities, and evaluate several state-of-the-art LALMs. Our results reveal substantial safety inconsistencies: different emotions elicit varying levels of unsafe responses, and the effect of intensity is non-monotonic, with medium expressions often posing the greatest risk. These findings highlight an overlooked vulnerability in LALMs and call for alignment strategies explicitly designed to ensure robustness under emotional variation, a prerequisite for trustworthy deployment in real-world settings.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 37 of its 39 references that are in the map. First posted on arXiv 19 Oct 2025.
Builds on
37 of 39 references.
References · 37
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2024GPT-4o System Cardcited by 4,467
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
- 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
- 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
- 2025Qwen2.5-Omni Technical Reportcited by 704
- 2024Qwen2-Audio Technical Reportcited by 543
- 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
- 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
- 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
25 more
- 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
- 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
- 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
- 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
- 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
- 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
- 2024Best-of-N Jailbreakingcited by 71
- 2024Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Taskscited by 70
- 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
- 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 43
- 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
- 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
- 2024Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attackscited by 35
- 2024Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Modelscited by 31
- 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
- 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
- 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
- 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 25
- 2024Building a Taiwanese Mandarin Spoken Language Model: A First Attemptcited by 22
- 2024Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Modelscited by 20
- 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
- 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
- 2025A Preliminary Exploration with GPT-4o Voice Modecited by 16
- 2025JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Modelscited by 11
- 2024Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Modelscited by 10
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Modelscited by 2
- 2026AOR-Bench: Do Large Audio Language Models Over-Refuse Pseudo-Harmful Queries?cited by 0
- 2026ARENA: Automated Red-Teaming for Large Audio Language Modelscited by 0
- 2026DuplexJail: Safety Alignment Breaks Under Spoken Interruption in Full-Duplex Modelscited by 0
- 2026DuplexJail: Safety Alignment Breaks Under Spoken Interruption in Full-Duplex Models8 Sep 2026
- 2026ARENA: Automated Red-Teaming for Large Audio Language Models16 Aug 2026
- 2026AOR-Bench: Do Large Audio Language Models Over-Refuse Pseudo-Harmful Queries?19 Jun 2026
- 2026MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models10 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2026A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook
- 2024Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insights
- 2026SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech
- 2025"I am bad": Interpreting Stealthy, Universal and Robust Audio Jailbreaks in Audio-Language Models
- 2026Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs
- 2025Synthetic Voices, Real Threats: Evaluating Large Text-to-Speech Models in Generating Harmful Audio
- 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts
- 2025Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guard