# Prompt-tuning in ASR systems for efficient domain-adaptation

arXiv: https://arxiv.org/abs/2110.06502 · HTML: https://mlsanity.com/paper/2110.06502/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 8 references that are in the map.
- First posted on arXiv 13 Oct 2021.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks](https://mlsanity.com/paper/1911.03863/) (2019)
- [Adapting Long Context NLM for ASR Rescoring in Conversational Agents](https://mlsanity.com/paper/2104.11070/) (2021)
- [ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modeling](https://mlsanity.com/paper/2106.09532/) (2021)

## Built on by (most cited first, 2 of 2)

- [Effective Transfer Learning for Low-Resource Natural Language Understanding](https://mlsanity.com/paper/2208.09180/) (2022)
- [A Better Start for Language Models: Domain-Conditional Position Offsets](https://mlsanity.com/paper/2607.18302/) (2026)

## Abstract (arXiv)

Automatic Speech Recognition (ASR) systems have found their use in numerous industrial applications in very diverse domains. Since domain-specific systems perform better than their generic counterparts on in-domain evaluation, the need for memory and compute-efficient domain adaptation is obvious. Particularly, adapting parameter-heavy transformer-based language models used for rescoring ASR hypothesis is challenging. In this work, we overcome the problem using prompt-tuning, a methodology that trains a small number of domain token embedding parameters to prime a transformer-based LM to a particular domain. With just a handful of extra parameters per domain, we achieve much better perplexity scores over the baseline of using an unadapted LM. Despite being parameter-efficient, these improvements are comparable to those of fully-fine-tuned models with hundreds of millions of parameters. We replicate our findings in perplexity numbers to Word Error Rate in a domain-specific ASR system for one such domain.