# Are Neural Open-Domain Dialog Systems Robust to Speech Recognition Errors in the Dialog History? An Empirical Study

arXiv: https://arxiv.org/abs/2008.07683 · HTML: https://mlsanity.com/paper/2008.07683/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2020.

## Built on by (most cited first, 19 of 19)

- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- ["How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversations](https://mlsanity.com/paper/2109.13489/) (2021)
- [Revisit Input Perturbation Problems for LLMs: A Unified Robustness Evaluation Framework for Noisy Slot Filling Task](https://mlsanity.com/paper/2310.06504/) (2023)
- [VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator](https://mlsanity.com/paper/2105.11589/) (2021)
- [TOD-DA: Towards Boosting the Robustness of Task-oriented Dialogue Modeling on Spoken Conversations](https://mlsanity.com/paper/2112.12441/) (2021)
- [Beyond Domain APIs: Task-oriented Conversational Modeling with Unstructured Knowledge Access Track in DSTC9](https://mlsanity.com/paper/2101.09276/) (2021)
- [PSSAT: A Perturbed Semantic Structure Awareness Transferring Method for Perturbation-Robust Slot Filling](https://mlsanity.com/paper/2208.11508/) (2022)
- [Measuring the Effect of Transcription Noise on Downstream Language Understanding Tasks](https://mlsanity.com/paper/2502.13645/) (2025)
- [Adapting Document-Grounded Dialog Systems to Spoken Conversations using Data Augmentation and a Noisy Channel Model](https://mlsanity.com/paper/2112.08844/) (2021)
- [Towards Robust and Generalizable Training: An Empirical Study of Noisy Slot Filling for Input Perturbations](https://mlsanity.com/paper/2310.03518/) (2023)
- [DemoNSF: A Multi-task Demonstration-based Generative Framework for Noisy Slot Filling Task](https://mlsanity.com/paper/2310.10169/) (2023)
- [Athena 2.0: Contextualized Dialogue Management for an Alexa Prize SocialBot](https://mlsanity.com/paper/2111.02519/) (2021)
- [A Systematic Evaluation of Response Selection for Open Domain Dialogue](https://mlsanity.com/paper/2208.04379/) (2022)
- [NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation](https://mlsanity.com/paper/2111.05196/) (2021)
- [Tokenwise Contrastive Pretraining for Finer Speech-to-BERT Alignment in End-to-End Speech-to-Intent Systems](https://mlsanity.com/paper/2204.05188/) (2022)
- [Unstructured Knowledge Access in Task-oriented Dialog Modeling using Language Inference, Knowledge Retrieval and Knowledge-Integrative Response Generation](https://mlsanity.com/paper/2101.06066/) (2021)
- [Towards Generalized Models for Task-oriented Dialogue Modeling on Spoken Conversations](https://mlsanity.com/paper/2203.04045/) (2022)
- [Interventional Speech Noise Injection for ASR Generalizable Spoken Language Understanding](https://mlsanity.com/paper/2410.15609/) (2024)
- [ReIn: Conversational Error Recovery with Reasoning Inception](https://mlsanity.com/paper/2602.17022/) (2026)

## Abstract (arXiv)

Large end-to-end neural open-domain chatbots are becoming increasingly popular. However, research on building such chatbots has typically assumed that the user input is written in nature and it is not clear whether these chatbots would seamlessly integrate with automatic speech recognition (ASR) models to serve the speech modality. We aim to bring attention to this important question by empirically studying the effects of various types of synthetic and actual ASR hypotheses in the dialog history on TransferTransfo, a state-of-the-art Generative Pre-trained Transformer (GPT) based neural open-domain dialog system from the NeurIPS ConvAI2 challenge. We observe that TransferTransfo trained on written data is very sensitive to such hypotheses introduced to the dialog history during inference time. As a baseline mitigation strategy, we introduce synthetic ASR hypotheses to the dialog history during training and observe marginal improvements, demonstrating the need for further research into techniques to make end-to-end open-domain chatbots fully speech-robust. To the best of our knowledge, this is the first study to evaluate the effects of synthetic and actual ASR hypotheses on a state-of-the-art neural open-domain dialog system and we hope it promotes speech-robustness as an evaluation criterion in open-domain dialog.