# Comparing Human and Machine Errors in Conversational Speech Transcription

arXiv: https://arxiv.org/abs/1708.08615 · HTML: https://mlsanity.com/paper/1708.08615/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2017.

## Built on by (most cited first, 7 of 7)

- [Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparities](https://mlsanity.com/paper/2207.11345/) (2022)
- [Arabic Speech Recognition by End-to-End, Modular Systems and Human](https://mlsanity.com/paper/2101.08454/) (2021)
- [The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systems](https://mlsanity.com/paper/2307.15493/) (2023)
- [MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder](https://mlsanity.com/paper/2409.14074/) (2024)
- [Toward Zero Oracle Word Error Rate on the Switchboard Benchmark](https://mlsanity.com/paper/2206.06192/) (2022)
- [Automatic Speech Recognition for Non-Native English: Accuracy and Disfluency Handling](https://mlsanity.com/paper/2503.06924/) (2025)
- [Human and Automatic Speech Recognition Performance on German Oral History Interviews](https://mlsanity.com/paper/2201.06841/) (2022)

## Abstract (arXiv)

Recent work in automatic recognition of conversational telephone speech (CTS) has achieved accuracy levels comparable to human transcribers, although there is some debate how to precisely quantify human performance on this task, using the NIST 2000 CTS evaluation set. This raises the question what systematic differences, if any, may be found differentiating human from machine transcription errors. In this paper we approach this question by comparing the output of our most accurate CTS recognition system to that of a standard speech transcription vendor pipeline. We find that the most frequent substitution, deletion and insertion error types of both outputs show a high degree of overlap. The only notable exception is that the automatic recognizer tends to confuse filled pauses ("uh") and backchannel acknowledgments ("uhhuh"). Humans tend not to make this error, presumably due to the distinctive and opposing pragmatic functions attached to these words. Furthermore, we quantify the correlation between human and machine errors at the speaker level, and investigate the effect of speaker overlap between training and test data. Finally, we report on an informal "Turing test" asking humans to discriminate between automatic and human transcription error cases.