# INTapt: Information-Theoretic Adversarial Prompt Tuning for Enhanced Non-Native Speech Recognition

arXiv: https://arxiv.org/abs/2305.16371 · HTML: https://mlsanity.com/paper/2305.16371/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security.
- Builds on 10 of its 22 references that are in the map.
- First posted on arXiv 25 May 2023.

## Placed in

- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 10 of 10)

- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo](https://mlsanity.com/paper/2203.17248/) (2022)
- [Joint Modeling of Accents and Acoustics for Multi-Accent Speech Recognition](https://mlsanity.com/paper/1802.02656/) (2018)
- [AIPNet: Generative Adversarial Pre-training of Accent-invariant Networks for End-to-end Speech Recognition](https://mlsanity.com/paper/1911.11935/) (2019)
- [Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English with Transfer Learning](https://mlsanity.com/paper/2110.00678/) (2021)
- [SMSMix: Sense-Maintained Sentence Mixup for Word Sense Disambiguation](https://mlsanity.com/paper/2212.07072/) (2022)

## Built on by (most cited first, 3 of 3)

- [C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion](https://mlsanity.com/paper/2403.14119/) (2024)
- [FRAG: Frequency Adapting Group for Diffusion Video Editing](https://mlsanity.com/paper/2406.06044/) (2024)
- [LI-TTA: Language Informed Test-Time Adaptation for Automatic Speech Recognition](https://mlsanity.com/paper/2408.05769/) (2024)

## Abstract (arXiv)

Automatic Speech Recognition (ASR) systems have attained unprecedented performance with large speech models pre-trained based on self-supervised speech representation learning. However, these pre-trained speech models suffer from representational bias as they tend to better represent those prominent accents (i.e., native (L1) English accent) in the pre-training speech corpus than less represented accents, resulting in a deteriorated performance for non-native (L2) English accents. Although there have been some approaches to mitigate this issue, all of these methods require updating the pre-trained model weights. In this paper, we propose Information Theoretic Adversarial Prompt Tuning (INTapt), which introduces prompts concatenated to the original input that can re-modulate the attention of the pre-trained model such that the corresponding input resembles a native (L1) English speech without updating the backbone weights. INTapt is trained simultaneously in the following two manners: (1) adversarial training to reduce accent feature dependence between the original input and the prompt-concatenated input and (2) training to minimize CTC loss for improving ASR performance to a prompt-concatenated input. Experimental results show that INTapt improves the performance of L2 English and increases feature similarity between L2 and L1 accents.