mlsanityData to 14 Oct 2026

Psychoacoustically Aligned Latent Smoothing for Adversarial Robustness of Full-Duplex Speech-to-Speech Dialogue Models

arXiv 2609.27378 ↗First posted 23 Sep 2026cs.SD cs.AI cs.CLcited by 0

On the map: LLM safety and security → Audio jailbreaks and speech backdoors

Abstract (arXiv)

End-to-end speech-to-speech dialogue models listen and speak simultaneously, so a continuously open acoustic channel is exposed to adversarial manipulation. We formalize imperceptible attacks on full-duplex agents as optimization over additive perturbations confined beneath the psychoacoustic masking threshold of the carrier speech, under three goals: targeted semantic hijacking, response suppression, and policy jailbreaking. Against an undefended Moshi-style agent, white-box attacks succeed in up to 91.7% of trials. We then introduce psychoacoustically aligned latent smoothing (PALS), which injects anisotropic Gaussian noise shaped by local codebook covariance at the residual-vector-quantized latent interface, with input noise shaped by the masking threshold constraining the attacker and trained by a Kullback--Leibler consistency objective. Deployed with no inference-time cost, PALS reduces hijack to 8.3%, mute to 11.2%, and jailbreak to 9.1% at clean quality within 2.3%. A Monte Carlo-smoothed variant certifies an ellipsoidal latent radius up to 0.616, a guaranteed floor that the empirical robustness far exceeds.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio jailbreaks and speech backdoors, one of 12 directions in LLM safety and security. Builds on 48 of its 59 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

48 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Models23 May
  2. 2024AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models11 Dec
  3. 2024Voice Jailbreak Attacks Against GPT-4o29 May
  4. 2024SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Models14 May

Other references · 44

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  2. 2013Intriguing properties of neural networkscited by 5,155
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  7. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  8. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  9. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  11. 2022High Fidelity Neural Audio Compressioncited by 687
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
32 more
  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  2. 2024Qwen2-Audio Technical Reportcited by 560
  3. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  5. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 468
  6. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  7. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  8. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  9. 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 330
  10. 2018Audio Adversarial Examples: Targeted Attacks on Speech-to-Textcited by 274
  11. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 224
  12. 2020Real Time Speech Enhancement in the Waveform Domaincited by 186
  13. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  14. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 172
  15. 2017Pyroomacoustics: A Python package for audio room simulations and array processing algorithmscited by 132
  16. 2022Generative Spoken Dialogue Language Modelingcited by 126
  17. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 100
  18. 2018Adversarial Attacks Against Automatic Speech Recognition Systems via Psychoacoustic Hidingcited by 85
  19. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 72
  20. 2020SoK: The Faults in our ASRs: An Overview of Attacks against Automatic Speech Recognition and Speaker Identification Systemscited by 46
  21. 2018Robust Audio Adversarial Example for a Physical Attackcited by 44
  22. 2019Universal Adversarial Perturbations for Speech Recognition Systemscited by 41
  23. 2021WaveGuard: Understanding and Mitigating Audio Adversarial Examplescited by 40
  24. 2025Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Datasetcited by 40
  25. 2020Data-Dependent Randomized Smoothingcited by 37
  26. 2021ANCER: Anisotropic Certification via Sample-wise Volume Maximizationcited by 23
  27. 2023Defending against Adversarial Audio via Diffusion Modelcited by 23
  28. 2024Speech Robust Bench: A Robustness Benchmark For Speech Recognitioncited by 21
  29. 2021Center Smoothing: Certified Robustness for Networks with Structured Outputscited by 18
  30. 2021Sequential Randomized Smoothing for Adversarially Robust Speech Recognitioncited by 13
  31. 2024Muting Whisper: A Universal Acoustic Adversarial Attack on Speech Foundation Modelscited by 12
  32. 2021Certified Defense via Latent Space Randomized Smoothing with Orthogonal Encoderscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Breaking Audio Large Language Models by Attacking Only the Encoder: A Universal Targeted Latent-Space Audio Attack
  2. 2026Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injection
  3. 2025"I am bad": Interpreting Stealthy, Universal and Robust Audio Jailbreaks in Audio-Language Models
  4. 2025Hidden in the Noise: Unveiling Backdoors in Audio LLMs Alignment through Latent Acoustic Pattern Triggers
  5. 2025When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputs
  6. 2026AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models
  7. 2024Noise-BERT: A Unified Perturbation-Robust Framework with Noise Alignment Pre-training for Noisy Slot Filling Task
  8. 2025ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio-Language Models