# Measuring the Effect of Transcription Noise on Downstream Language Understanding Tasks

arXiv: https://arxiv.org/abs/2502.13645 · HTML: https://mlsanity.com/paper/2502.13645/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 68 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Builds on (most cited first, 25 of 25)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://mlsanity.com/paper/2306.17563/) (2023)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension](https://mlsanity.com/paper/1804.00320/) (2018)
- [SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech](https://mlsanity.com/paper/2111.10367/) (2021)
- [A spelling correction model for end-to-end speech recognition](https://mlsanity.com/paper/1902.07178/) (2019)
- [SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks](https://mlsanity.com/paper/2212.10525/) (2022)
- [Robustness Testing of Language Understanding in Task-Oriented Dialog](https://mlsanity.com/paper/2012.15262/) (2020)
- [Knowledge Distillation for Improved Accuracy in Spoken Question Answering](https://mlsanity.com/paper/2010.11067/) (2020)
- ["How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversations](https://mlsanity.com/paper/2109.13489/) (2021)
- [How Bad Are Artifacts?: Analyzing the Impact of Speech Enhancement Errors on ASR](https://mlsanity.com/paper/2201.06685/) (2022)
- [FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition](https://mlsanity.com/paper/2109.14420/) (2021)
- [Are Neural Open-Domain Dialog Systems Robust to Speech Recognition Errors in the Dialog History? An Empirical Study](https://mlsanity.com/paper/2008.07683/) (2020)
- [Error Correction in ASR using Sequence-to-Sequence Models](https://mlsanity.com/paper/2202.01157/) (2022)
- [QAConv: Question Answering on Informative Conversations](https://mlsanity.com/paper/2105.06912/) (2021)
- [Robust Neural Machine Translation for Clean and Noisy Speech Transcripts](https://mlsanity.com/paper/1910.10238/) (2019)
- [Useful Blunders: Can Automated Speech Recognition Errors Improve Downstream Dementia Classification?](https://mlsanity.com/paper/2401.05551/) (2024)
- [Impact of ASR on Alzheimer's Disease Detection: All Errors are Equal, but Deletions are More Equal than Others](https://mlsanity.com/paper/1904.01684/) (2019)
- [An Approach to Improve Robustness of NLP Systems against ASR Errors](https://mlsanity.com/paper/2103.13610/) (2021)
- [Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors](https://mlsanity.com/paper/2310.16609/) (2023)
- [UCorrect: An Unsupervised Framework for Automatic Speech Recognition Error Correction](https://mlsanity.com/paper/2401.05689/) (2024)
- [Interventional Speech Noise Injection for ASR Generalizable Spoken Language Understanding](https://mlsanity.com/paper/2410.15609/) (2024)

## Built on by (most cited first, 7 of 7)

- [LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families](https://mlsanity.com/paper/2603.20042/) (2026)
- [When Misinformation Speaks and Converses: Rethinking Fact-Checking in Audio Platforms](https://mlsanity.com/paper/2604.16767/) (2026)
- [CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agents](https://mlsanity.com/paper/2606.21453/) (2026)
- [When Simpler Is Better: Evaluating Translation Pipelines for Medieval Latin Manuscripts](https://mlsanity.com/paper/2607.03836/) (2026)
- [Whisper-Based Speech Transcription from Videos Across Multiple Languages for Cross-Cultural Understanding](https://mlsanity.com/paper/2609.11772/) (2026)
- [CoSE-E: A Benchmark for Code-switched Speech Evaluation in Enterprise Settings](https://mlsanity.com/paper/2609.35645/) (2026)
- [When Does an ASR Error Matter? Task-Role Diagnostics for Tool-Using Voice Agents](https://mlsanity.com/paper/2610.17619/) (2026)

## Abstract (arXiv)

With the increasing prevalence of recorded human speech, spoken language understanding (SLU) is essential for its efficient processing. In order to process the speech, it is commonly transcribed using automatic speech recognition technology. This speech-to-text transition introduces errors into the transcripts, which subsequently propagate to downstream NLP tasks, such as dialogue summarization. While it is known that transcript noise affects downstream tasks, a systematic approach to analyzing its effects across different noise severities and types has not been addressed. We propose a configurable framework for assessing task models in diverse noisy settings, and for examining the impact of transcript-cleaning techniques. The framework facilitates the investigation of task model behavior, which can in turn support the development of effective SLU solutions. We exemplify the utility of our framework on three SLU tasks and four task models, offering insights regarding the effect of transcript noise on tasks in general and models in particular. For instance, we find that task models can tolerate a certain level of noise, and are affected differently by the types of errors in the transcript.