# Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors

arXiv: https://arxiv.org/abs/2310.16609 · HTML: https://mlsanity.com/paper/2310.16609/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages](https://mlsanity.com/paper/2204.08582/) (2022)
- [RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentation](https://mlsanity.com/paper/1905.03072/) (2019)
- [Listening while Speaking: Speech Chain by Deep Learning](https://mlsanity.com/paper/1707.04879/) (2017)
- [RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems](https://mlsanity.com/paper/2012.14666/) (2020)
- [Robustness Testing of Language Understanding in Task-Oriented Dialog](https://mlsanity.com/paper/2012.15262/) (2020)
- [You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation](https://mlsanity.com/paper/2005.07157/) (2020)
- [Back-Translation-Style Data Augmentation for End-to-End ASR](https://mlsanity.com/paper/1807.10893/) (2018)
- [fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit](https://mlsanity.com/paper/2109.06912/) (2021)
- [On the Robustness of Intent Classification and Slot Labeling in Goal-oriented Dialog Systems to Real-world Noise](https://mlsanity.com/paper/2104.07149/) (2021)
- [Towards a Competitive End-to-End Speech Recognition for CHiME-6 Dinner Party Transcription](https://mlsanity.com/paper/2004.10799/) (2020)

## Built on by (most cited first, 6 of 6)

- [reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs](https://mlsanity.com/paper/2503.11751/) (2025)
- [Measuring the Effect of Transcription Noise on Downstream Language Understanding Tasks](https://mlsanity.com/paper/2502.13645/) (2025)
- [A Three-Pronged Approach to Cross-Lingual Adaptation with Multilingual LLMs](https://mlsanity.com/paper/2406.17377/) (2024)
- [k* Distribution: Evaluating the Latent Space of Deep Neural Networks using Local Neighborhood Analysis](https://mlsanity.com/paper/2312.04024/) (2023)
- [Preservation of Language Understanding Capabilities in Speech-aware Large Language Models](https://mlsanity.com/paper/2509.12171/) (2025)
- [WASIL: In-the-Wild Arabic Spoken Interactions with LLMs](https://mlsanity.com/paper/2605.16364/) (2026)

## Abstract (arXiv)

In a spoken dialogue system, an NLU model is preceded by a speech recognition system that can deteriorate the performance of natural language understanding. This paper proposes a method for investigating the impact of speech recognition errors on the performance of natural language understanding models. The proposed method combines the back transcription procedure with a fine-grained technique for categorizing the errors that affect the performance of NLU models. The method relies on the usage of synthesized speech for NLU evaluation. We show that the use of synthesized speech in place of audio recording does not change the outcomes of the presented technique in a significant way.