# Hear No Evil: Towards Adversarial Robustness of Automatic Speech Recognition via Multi-Task Learning

arXiv: https://arxiv.org/abs/2204.02381 · HTML: https://mlsanity.com/paper/2204.02381/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 5 Apr 2022.

## Builds on (most cited first, 23 of 23)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Audio Adversarial Examples: Targeted Attacks on Speech-to-Text](https://mlsanity.com/paper/1801.01944/) (2018)
- [Adversarial Robustness Toolbox v1.0.0](https://mlsanity.com/paper/1807.01069/) (2018)
- [CommanderSong: A Systematic Approach for Practical Adversarial Voice Recognition](https://mlsanity.com/paper/1801.08535/) (2018)
- [A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks](https://mlsanity.com/paper/2010.11338/) (2020)
- [Houdini: Fooling Deep Structured Prediction Models](https://mlsanity.com/paper/1707.05373/) (2017)
- [Did you hear that? Adversarial Examples Against Automatic Speech Recognition](https://mlsanity.com/paper/1801.00554/) (2018)
- [Targeted Adversarial Examples for Black Box Audio Systems](https://mlsanity.com/paper/1805.07820/) (2018)
- [Domain Adversarial Training for Accented Speech Recognition](https://mlsanity.com/paper/1806.02786/) (2018)
- [WaveGuard: Understanding and Mitigating Audio Adversarial Examples](https://mlsanity.com/paper/2103.03344/) (2021)
- [Why does CTC result in peaky behavior?](https://mlsanity.com/paper/2105.14849/) (2021)
- [Adversarial Attacks and Defenses for Speech Recognition Systems](https://mlsanity.com/paper/2103.17122/) (2021)
- [Characterizing Speech Adversarial Examples Using Self-Attention U-Net Enhancement](https://mlsanity.com/paper/2003.13917/) (2020)
- [Best of Both Worlds: Robust Accented Speech Recognition with Adversarial Transfer Learning](https://mlsanity.com/paper/2103.05834/) (2021)
- [Adversarial Robustness in Multi-Task Learning: Promises and Illusions](https://mlsanity.com/paper/2110.15053/) (2021)
- [ADAGIO: Interactive Experimentation with Adversarial Attack and Defense for Audio](https://mlsanity.com/paper/1805.11852/) (2018)
- [To Reverse the Gradient or Not: An Empirical Comparison of Adversarial and Multi-task Learning in Speech Recognition](https://mlsanity.com/paper/1812.03483/) (2018)
- [Adversarial Joint Training with Self-Attention Mechanism for Robust End-to-End Speech Recognition](https://mlsanity.com/paper/2104.01471/) (2021)
- [Perceptual-based deep-learning denoiser as a defense against adversarial attacks on ASR systems](https://mlsanity.com/paper/2107.05222/) (2021)

## Abstract (arXiv)

As automatic speech recognition (ASR) systems are now being widely deployed in the wild, the increasing threat of adversarial attacks raises serious questions about the security and reliability of using such systems. On the other hand, multi-task learning (MTL) has shown success in training models that can resist adversarial attacks in the computer vision domain. In this work, we investigate the impact of performing such multi-task learning on the adversarial robustness of ASR models in the speech domain. We conduct extensive MTL experimentation by combining semantically diverse tasks such as accent classification and ASR, and evaluate a wide range of adversarial settings. Our thorough analysis reveals that performing MTL with semantically diverse tasks consistently makes it harder for an adversarial attack to succeed. We also discuss in detail the serious pitfalls and their related remedies that have a significant impact on the robustness of MTL models. Our proposed MTL approach shows considerable absolute improvements in adversarially targeted WER ranging from 17.25 up to 59.90 compared to single-task learning baselines (attention decoder and CTC respectively). Ours is the first in-depth study that uncovers adversarial robustness gains from multi-task learning for ASR.