# Joint Contextual Modeling for ASR Correction and Language Understanding

arXiv: https://arxiv.org/abs/2002.00750 · HTML: https://mlsanity.com/paper/2002.00750/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2020.

## Built on by (most cited first, 30 of 30)

- [N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space](https://mlsanity.com/paper/2303.00456/) (2023)
- [Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2109.00627/) (2021)
- ["How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversations](https://mlsanity.com/paper/2109.13489/) (2021)
- [Phoneme-BERT: Joint Language Modelling of Phoneme Sequence and ASR Transcript](https://mlsanity.com/paper/2102.00804/) (2021)
- [BART based semantic correction for Mandarin automatic speech recognition system](https://mlsanity.com/paper/2104.05507/) (2021)
- [Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognition](https://mlsanity.com/paper/2207.00857/) (2022)
- [Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systems](https://mlsanity.com/paper/2112.05842/) (2021)
- [ASR-GLUE: A New Multi-task Benchmark for ASR-Robust Natural Language Understanding](https://mlsanity.com/paper/2108.13048/) (2021)
- [PSSAT: A Perturbed Semantic Structure Awareness Transferring Method for Perturbation-Robust Slot Filling](https://mlsanity.com/paper/2208.11508/) (2022)
- [SoftCorrect: Error Correction with Soft Detection for Automatic Speech Recognition](https://mlsanity.com/paper/2212.01039/) (2022)
- [An Approach to Improve Robustness of NLP Systems against ASR Errors](https://mlsanity.com/paper/2103.13610/) (2021)
- [Correcting Automated and Manual Speech Transcription Errors using Warped Language Models](https://mlsanity.com/paper/2103.14580/) (2021)
- [Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling](https://mlsanity.com/paper/2205.12986/) (2022)
- [Cross-Modal ASR Post-Processing System for Error Correction and Utterance Rejection](https://mlsanity.com/paper/2201.03313/) (2022)
- [ASR Error Correction with Constrained Decoding on Operation Prediction](https://mlsanity.com/paper/2208.04641/) (2022)
- [Contextual ASR Error Handling with LLMs Augmentation for Goal-Oriented Conversational AI](https://mlsanity.com/paper/2501.06129/) (2025)
- [Adapting Document-Grounded Dialog Systems to Spoken Conversations using Data Augmentation and a Noisy Channel Model](https://mlsanity.com/paper/2112.08844/) (2021)
- [Unsupervised domain adaptation for speech recognition with unsupervised error correction](https://mlsanity.com/paper/2209.12043/) (2022)
- [The Eloquence team submission for task 1 of MLC-SLM challenge](https://mlsanity.com/paper/2507.19308/) (2025)
- [A Token-Wise Beam Search Algorithm for RNN-T](https://mlsanity.com/paper/2302.14357/) (2023)
- [Knowledge-Aware Audio-Grounded Generative Slot Filling for Limited Annotated Data](https://mlsanity.com/paper/2307.01764/) (2023)
- [MathSpeech: Leveraging Small LMs for Accurate Conversion in Mathematical Speech-to-Formula](https://mlsanity.com/paper/2412.15655/) (2024)
- [Towards Understanding Omission in Dialogue Summarization](https://mlsanity.com/paper/2211.07145/) (2022)
- [End-to-End Speech to Intent Prediction to improve E-commerce Customer Support Voicebot in Hindi and English](https://mlsanity.com/paper/2211.07710/) (2022)
- [Enhancing conversational quality in language learning chatbots: An evaluation of GPT4 for ASR error correction](https://mlsanity.com/paper/2307.09744/) (2023)
- [Robust Unstructured Knowledge Access in Conversational Dialogue with ASR Errors](https://mlsanity.com/paper/2211.03990/) (2022)
- [A Pilot Study of Applying Sequence-to-Sequence Voice Conversion to Evaluate the Intelligibility of L2 Speech Using a Native Speaker's Shadowings](https://mlsanity.com/paper/2410.02239/) (2024)
- [Improving Distinction between ASR Errors and Speech Disfluencies with Feature Space Interpolation](https://mlsanity.com/paper/2108.01812/) (2021)
- [Building Robust Spoken Language Understanding by Cross Attention between Phoneme Sequence and ASR Hypothesis](https://mlsanity.com/paper/2203.12067/) (2022)
- [CoSE-E: A Benchmark for Code-switched Speech Evaluation in Enterprise Settings](https://mlsanity.com/paper/2609.35645/) (2026)

## Abstract (arXiv)

The quality of automatic speech recognition (ASR) is critical to Dialogue Systems as ASR errors propagate to and directly impact downstream tasks such as language understanding (LU). In this paper, we propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with LU to improve the performance of both tasks simultaneously. To measure the effectiveness of this approach we used a public benchmark, the 2nd Dialogue State Tracking (DSTC2) corpus. As a baseline approach, we trained task-specific Statistical Language Models (SLM) and fine-tuned state-of-the-art Generalized Pre-training (GPT) Language Model to re-rank the n-best ASR hypotheses, followed by a model to identify the dialog act and slots. i) We further trained ranker models using GPT and Hierarchical CNN-RNN models with discriminatory losses to detect the best output given n-best hypotheses. We extended these ranker models to first select the best ASR output and then identify the dialogue act and slots in an end to end fashion. ii) We also proposed a novel joint ASR error correction and LU model, a word confusion pointer network (WCN-Ptr) with multi-head self-attention on top, which consumes the word confusions populated from the n-best. We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.