mlsanityData to 7 Oct 2026

SPIRIT: Patching Speech Language Models against Jailbreak Attacks

arXiv 2505.13541 ↗First posted 18 May 2025eess.AS cs.LGcited by 7

On the map: LLM safety and security → Audio jailbreaks and speech backdoors

Abstract (arXiv)

Speech Language Models (SLMs) enable natural interactions via spoken instructions, which more effectively capture user intent by detecting nuances in speech. The richer speech signal introduces new security risks compared to text-based models, as adversaries can better bypass safety mechanisms by injecting imperceptible noise to speech. We analyze adversarial attacks and find that SLMs are substantially more vulnerable to jailbreak attacks, which can achieve a perfect 100% attack success rate in some instances. To improve security, we propose post-hoc patching defenses used to intervene during inference by modifying the SLM's activations that improve robustness up to 99% with (i) negligible impact on utility and (ii) without any re-training. We conduct ablation studies to maximize the efficacy of our defenses and improve the utility/security trade-off, validated with large-scale benchmarks unique to SLMs.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Audio jailbreaks and speech backdoors, one of 12 directions in LLM safety and security. Builds on 32 of its 45 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

32 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models11 Dec

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
19 more
  1. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  2. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  3. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  4. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  5. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  6. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  7. 2021Causal Abstractions of Neural Networkscited by 401
  8. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  9. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  10. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  11. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  12. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  13. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
  14. 2021Adversarial Robustness with Semi-Infinite Constrained Learningcited by 39
  15. 2024Dissecting Language Models: Machine Unlearning via Selective Pruningcited by 38
  16. 2022Probabilistically Robust Learning: Balancing Average- and Worst-case Performancecited by 36
  17. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  18. 2024Repetition Neurons: How Do Language Models Produce Repetitions?cited by 22
  19. 2025SparQLe: Speech Queries to Text Translation Through LLMscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025ERIS: Evolutionary Real-world Interference Scheme for Jailbreaking Audio Large Modelscited by 2
  2. 2025SmoothGuard: Defending Multimodal Large Language Models with Noise Perturbation and Clustering Aggregationcited by 2
  3. 2025Rebellion: Noise-Robust Reasoning Training for Audio Reasoning Modelscited by 1
  4. 2026SoundBreak: A Systematic Study of Audio-Only Adversarial Attacks on Trimodal Modelscited by 1
  5. 2026Resurfacing Paralinguistic Awareness in Large Audio Language Modelscited by 1
  6. 2026Audio Jailbreaks in Large Audio-Language Models: Taxonomy, Attack-Defense Analysis, and Cost-Aware Evaluationcited by 0
  7. 2026Causal Localization of the Refusal Direction in Audio Language Modelscited by 0