# ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modeling

arXiv: https://arxiv.org/abs/2106.09532 · HTML: https://mlsanity.com/paper/2106.09532/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context](https://mlsanity.com/paper/1805.04623/) (2018)
- [On the Sentence Embeddings from Pre-trained Language Models](https://mlsanity.com/paper/2011.05864/) (2020)
- [The Microsoft 2017 Conversational Speech Recognition System](https://mlsanity.com/paper/1708.06073/) (2017)
- [Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces](https://mlsanity.com/paper/2008.06173/) (2020)
- [Multi-task Language Modeling for Improving Speech Recognition of Rare Words](https://mlsanity.com/paper/2011.11715/) (2020)
- [Joint Online Spoken Language Understanding and Language Modeling with Recurrent Neural Networks](https://mlsanity.com/paper/1609.01462/) (2016)
- [Multimodal Semi-supervised Learning Framework for Punctuation Prediction in Conversational Speech](https://mlsanity.com/paper/2008.00702/) (2020)
- [Transformer Language Models with LSTM-based Cross-utterance Information Representation](https://mlsanity.com/paper/2102.06474/) (2021)
- [Adapting Long Context NLM for ASR Rescoring in Conversational Agents](https://mlsanity.com/paper/2104.11070/) (2021)
- [Scalable language model adaptation for spoken dialogue systems](https://mlsanity.com/paper/1812.04647/) (2018)
- [Contextualizing ASR Lattice Rescoring with Hybrid Pointer Network Language Model](https://mlsanity.com/paper/2005.07394/) (2020)
- [Contextual Language Model Adaptation for Conversational Agents](https://mlsanity.com/paper/1806.10215/) (2018)
- [Long-span language modeling for speech recognition](https://mlsanity.com/paper/1911.04571/) (2019)

## Built on by (most cited first, 7 of 7)

- [Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition](https://mlsanity.com/paper/2306.16007/) (2023)
- [An Integrative Survey on Mental Health Conversational Agents to Bridge Computer Science and Medical Perspectives](https://mlsanity.com/paper/2310.17017/) (2023)
- [Lumos : Empowering Multimodal LLMs with Scene Text Recognition](https://mlsanity.com/paper/2402.08017/) (2024)
- [BERT Attends the Conversation: Improving Low-Resource Conversational ASR](https://mlsanity.com/paper/2110.02267/) (2021)
- [Prompt-tuning in ASR systems for efficient domain-adaptation](https://mlsanity.com/paper/2110.06502/) (2021)
- [Prompt Tuning GPT-2 language model for parameter-efficient domain adaptation of ASR systems](https://mlsanity.com/paper/2112.08718/) (2021)
- [Mask The Bias: Improving Domain-Adaptive Generalization of CTC-based ASR with Internal Language Model Estimation](https://mlsanity.com/paper/2305.03837/) (2023)

## Abstract (arXiv)

Automatic Speech Recognition (ASR) robustness toward slot entities are critical in e-commerce voice assistants that involve monetary transactions and purchases. Along with effective domain adaptation, it is intuitive that cross utterance contextual cues play an important role in disambiguating domain specific content words from speech. In this paper, we investigate various techniques to improve contextualization, content word robustness and domain adaptation of a Transformer-XL neural language model (NLM) to rescore ASR N-best hypotheses. To improve contextualization, we utilize turn level dialogue acts along with cross utterance context carry over. Additionally, to adapt our domain-general NLM towards e-commerce on-the-fly, we use embeddings derived from a finetuned masked LM on in-domain data. Finally, to improve robustness towards in-domain content words, we propose a multi-task model that can jointly perform content word detection and language modeling tasks. Compared to a non-contextual LSTM LM baseline, our best performing NLM rescorer results in a content WER reduction of 19.2% on e-commerce audio test set and a slot labeling F1 improvement of 6.4%.