# Characterizing Speech Adversarial Examples Using Self-Attention U-Net Enhancement

arXiv: https://arxiv.org/abs/2003.13917 · HTML: https://mlsanity.com/paper/2003.13917/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2020.

## Built on by (most cited first, 18 of 18)

- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [The Dawn of Quantum Natural Language Processing](https://mlsanity.com/paper/2110.06510/) (2021)
- [A Study on Speech Enhancement Based on Diffusion Probabilistic Model](https://mlsanity.com/paper/2107.11876/) (2021)
- [Classical-to-Quantum Transfer Learning for Spoken Command Recognition Based on Quantum Neural Networks](https://mlsanity.com/paper/2110.08689/) (2021)
- [Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"](https://mlsanity.com/paper/2105.14538/) (2021)
- [Voting for the right answer: Adversarial defense for speaker verification](https://mlsanity.com/paper/2106.07868/) (2021)
- [PATE-AAE: Incorporating Adversarial Autoencoder into Private Aggregation of Teacher Ensembles for Spoken Command Classification](https://mlsanity.com/paper/2104.01271/) (2021)
- [LAFFNet: A Lightweight Adaptive Feature Fusion Network for Underwater Image Enhancement](https://mlsanity.com/paper/2105.01299/) (2021)
- [SoK: A Modularized Approach to Study the Security of Automatic Speech Recognition Systems](https://mlsanity.com/paper/2103.10651/) (2021)
- [A Lottery Ticket Hypothesis Framework for Low-Complexity Device-Robust Neural Acoustic Scene Classification](https://mlsanity.com/paper/2107.01461/) (2021)
- [Mitigating Closed-model Adversarial Examples with Bayesian Neural Modeling for Enhanced End-to-End Speech Recognition](https://mlsanity.com/paper/2202.08532/) (2022)
- [NEUROSEC: FPGA-Based Neuromorphic Audio Security](https://mlsanity.com/paper/2401.12055/) (2024)
- [Robust Unsupervised Multi-Object Tracking in Noisy Environments](https://mlsanity.com/paper/2105.10005/) (2021)
- [Exploiting Hybrid Models of Tensor-Train Networks for Spoken Command Recognition](https://mlsanity.com/paper/2201.10609/) (2022)
- [MIXPGD: Hybrid Adversarial Training for Speech Recognition Systems](https://mlsanity.com/paper/2303.05758/) (2023)
- [Robust Automatic Speech Recognition via WavAugment Guided Phoneme Adversarial Training](https://mlsanity.com/paper/2307.12498/) (2023)
- [Investigating Training Strategies and Model Robustness of Low-Rank Adaptation for Language Modeling in Speech Recognition](https://mlsanity.com/paper/2401.10447/) (2024)
- [Hear No Evil: Towards Adversarial Robustness of Automatic Speech Recognition via Multi-Task Learning](https://mlsanity.com/paper/2204.02381/) (2022)

## Abstract (arXiv)

Recent studies have highlighted adversarial examples as ubiquitous threats to the deep neural network (DNN) based speech recognition systems. In this work, we present a U-Net based attention model, U-NetAt, to enhance adversarial speech signals. Specifically, we evaluate the model performance by interpretable speech recognition metrics and discuss the model performance by the augmented adversarial training. Our experiments show that our proposed U-NetAt improves the perceptual evaluation of speech quality (PESQ) from 1.13 to 2.78, speech transmission index (STI) from 0.65 to 0.75, short-term objective intelligibility (STOI) from 0.83 to 0.96 on the task of speech enhancement with adversarial speech examples. We conduct experiments on the automatic speech recognition (ASR) task with adversarial audio attacks. We find that (i) temporal features learned by the attention network are capable of enhancing the robustness of DNN based ASR models; (ii) the generalization power of DNN based ASR model could be enhanced by applying adversarial training with an additive adversarial data augmentation. The ASR metric on word-error-rates (WERs) shows that there is an absolute 2.22 % decrease under gradient-based perturbation, and an absolute 2.03 % decrease, under evolutionary-optimized perturbation, which suggests that our enhancement models with adversarial training can further secure a resilient ASR system.