# Promptformer: Prompted Conformer Transducer for ASR

arXiv: https://arxiv.org/abs/2401.07360 · HTML: https://mlsanity.com/paper/2401.07360/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 14 Jan 2024.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [HyperPrompt: Prompt-based Task-Conditioning of Transformers](https://mlsanity.com/paper/2203.00759/) (2022)
- [Contextual Adapters for Personalized Speech Recognition in Neural Transducers](https://mlsanity.com/paper/2205.13660/) (2022)
- [Context-Aware Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2111.03250/) (2021)
- [Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers](https://mlsanity.com/paper/2104.09426/) (2021)
- [Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillation](https://mlsanity.com/paper/2102.07935/) (2021)
- [PROCTER: PROnunciation-aware ConTextual adaptER for personalized speech recognition in neural transducers](https://mlsanity.com/paper/2303.17131/) (2023)
- [Towards End-to-End Integration of Dialog History for Improved Spoken Language Understanding](https://mlsanity.com/paper/2204.05169/) (2022)
- [Attentive Contextual Carryover for Multi-Turn End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2112.06743/) (2021)
- [Improving Transformer-based Conversational ASR by Inter-Sentential Attention Mechanism](https://mlsanity.com/paper/2207.00883/) (2022)
- [Dialog act guided contextual adapter for personalized speech recognition](https://mlsanity.com/paper/2303.17799/) (2023)
- [Joint Modelling of Spoken Language Understanding Tasks with Integrated Dialog History](https://mlsanity.com/paper/2305.00926/) (2023)
- [Contextual-Utterance Training for Automatic Speech Recognition](https://mlsanity.com/paper/2210.16238/) (2022)

## Built on by (most cited first, 3 of 3)

- [Improving Domain-Specific ASR with LLM-Generated Contextual Descriptions](https://mlsanity.com/paper/2407.17874/) (2024)
- [Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrieval](https://mlsanity.com/paper/2411.00664/) (2024)
- [An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems](https://mlsanity.com/paper/2409.10515/) (2024)

## Abstract (arXiv)

Context cues carry information which can improve multi-turn interactions in automatic speech recognition (ASR) systems. In this paper, we introduce a novel mechanism inspired by hyper-prompting to fuse textual context with acoustic representations in the attention mechanism. Results on a test set with multi-turn interactions show that our method achieves 5.9% relative word error rate reduction (rWERR) over a strong baseline. We show that our method does not degrade in the absence of context and leads to improvements even if the model is trained without context. We further show that leveraging a pre-trained sentence-piece model for context embedding generation can outperform an external BERT model.