# Contextual ASR Error Handling with LLMs Augmentation for Goal-Oriented Conversational AI

arXiv: https://arxiv.org/abs/2501.06129 · HTML: https://mlsanity.com/paper/2501.06129/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 40 references that are in the map.
- First posted on arXiv 10 Jan 2025.

## Builds on (most cited first, 11 of 11)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [Joint Contextual Modeling for ASR Correction and Language Understanding](https://mlsanity.com/paper/2002.00750/) (2020)
- [Feedback-Based Self-Learning in Large-Scale Conversational AI Agents](https://mlsanity.com/paper/1911.02557/) (2019)
- [D-CALM: A Dynamic Clustering-based Active Learning Approach for Mitigating Bias](https://mlsanity.com/paper/2305.17013/) (2023)
- [Learning from Past Mistakes: Improving Automatic Speech Recognition Output via Noisy-Clean Phrase Context Modeling](https://mlsanity.com/paper/1802.02607/) (2018)
- [CASA-NLU: Context-Aware Self-Attentive Natural Language Understanding for Task-Oriented Chatbots](https://mlsanity.com/paper/1909.08705/) (2019)
- [Remember the context! ASR slot error correction through memorization](https://mlsanity.com/paper/2109.05092/) (2021)

## Built on by (most cited first, 6 of 6)

- [Contextualized Token Discrimination for Speech Search Query Correction](https://mlsanity.com/paper/2509.04393/) (2025)
- [Semantic-Augmented Latent Topic Modeling with LLM-in-the-Loop](https://mlsanity.com/paper/2507.08498/) (2025)
- [Utterance-Level Methods for Identifying Reliable ASR-Output for Child Speech](https://mlsanity.com/paper/2604.19801/) (2026)
- [Proactive for Uncertainty: Cause-Aware Error Diagnosis and Interactive Clarification for Spoken Dialogue Systems](https://mlsanity.com/paper/2605.25404/) (2026)
- [Moonworks Lunara: Modeling Artistic Intelligence](https://mlsanity.com/paper/2609.22272/) (2026)
- [Select or Rewrite: Oracle-Guided Fine-Tuning for Generative ASR Post-Editing from N-Best Hypotheses](https://mlsanity.com/paper/2610.15056/) (2026)

## Abstract (arXiv)

General-purpose automatic speech recognition (ASR) systems do not always perform well in goal-oriented dialogue. Existing ASR correction methods rely on prior user data or named entities. We extend correction to tasks that have no prior user data and exhibit linguistic flexibility such as lexical and syntactic variations. We propose a novel context augmentation with a large language model and a ranking strategy that incorporates contextual information from the dialogue states of a goal-oriented conversational AI and its tasks. Our method ranks (1) n-best ASR hypotheses by their lexical and semantic similarity with context and (2) context by phonetic correspondence with ASR hypotheses. Evaluated in home improvement and cooking domains with real-world users, our method improves recall and F1 of correction by 34% and 16%, respectively, while maintaining precision and false positive rate. Users rated .8-1 point (out of 5) higher when our correction method worked properly, with no decrease due to false positives.