mlsanityData to 14 Oct 2026

Reshaping Representation Space to Balance the Safety and Over-rejection in Large Audio Language Models

arXiv 2505.19670 ↗First posted 26 May 2025cs.CL cs.MM cs.SDcited by 7

Not on the map. Why?

Abstract (arXiv)

Large Audio Language Models (LALMs) have extended the capabilities of Large Language Models (LLMs) by enabling audio-based human interactions. However, recent research has revealed that LALMs remain vulnerable to harmful queries due to insufficient safety-alignment. Despite advances in defence measures for text and vision LLMs, effective safety-alignment strategies and audio-safety dataset specifically targeting LALMs are notably absent. Meanwhile defence measures based on Supervised Fine-tuning (SFT) struggle to address safety improvement while avoiding over-rejection issues, significantly compromising helpfulness. In this work, we propose an unsupervised safety-fine-tuning strategy as remedy that reshapes model's representation space to enhance existing LALMs safety-alignment while balancing the risk of over-rejection. Our experiments, conducted across three generations of Qwen LALMs, demonstrate that our approach significantly improves LALMs safety under three modality input conditions (audio-text, text-only, and audio-only) while increasing over-rejection rate by only 0.88% on average. Warning: this paper contains harmful examples.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 18 of its 33 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

18 of 33 references.

References · 18

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  4. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  5. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  6. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  7. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  8. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
  9. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  10. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 92
  11. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 74
  12. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 72
6 more
  1. 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
  2. 2024Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Modelscited by 37
  3. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
  4. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  5. 2024MoGU: A Framework for Enhancing Safety of Open-Sourced LLMs While Preserving Their Usabilitycited by 15
  6. 2024Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insightscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2025SARSteer: Safeguarding Large Audio-Language Models via Safe-Ablated Refusal Steeringcited by 8
  2. 2026Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignmentcited by 6
  3. 2026Resurfacing Paralinguistic Awareness in Large Audio Language Modelscited by 2
  4. 2026Audio Jailbreaks in Large Audio-Language Models: Taxonomy, Attack-Defense Analysis, and Cost-Aware Evaluationcited by 1
  5. 2026AEGIS: Audio Endogenous Guarding via Internal Signals Against Large Audio-Language Model Jailbreakscited by 1
  6. 2026Causal Localization of the Refusal Direction in Audio Language Modelscited by 0
  7. 2026From Acoustic Sensing to Physical Actions: A Comprehensive Survey of Security Risks in Edge-Deployed Audio LLM-Based Agentscited by 0