# English Conversational Telephone Speech Recognition by Humans and Machines

arXiv: https://arxiv.org/abs/1703.02136 · HTML: https://mlsanity.com/paper/1703.02136/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2017.

## Built on by (most cited first, 30 of 40)

- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [MARS: Unleashing the Power of Variance Reduction for Training Large Models](https://mlsanity.com/paper/2411.10438/) (2024)
- [QASR: QCRI Aljazeera Speech Resource -- A Large Scale Annotated Arabic Speech Corpus](https://mlsanity.com/paper/2106.13000/) (2021)
- [On the limit of English conversational speech recognition](https://mlsanity.com/paper/2105.00982/) (2021)
- [Investigation of Data Augmentation Techniques for Disordered Speech Recognition](https://mlsanity.com/paper/2201.05562/) (2022)
- [Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systems](https://mlsanity.com/paper/2204.00212/) (2022)
- [Measuring the Accuracy of Automatic Speech Recognition Solutions](https://mlsanity.com/paper/2408.16287/) (2024)
- [Arabic Speech Recognition by End-to-End, Modular Systems and Human](https://mlsanity.com/paper/2101.08454/) (2021)
- [PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations](https://mlsanity.com/paper/2203.16965/) (2022)
- [The History of Speech Recognition to the Year 2030](https://mlsanity.com/paper/2108.00084/) (2021)
- [Platform for Situated Intelligence](https://mlsanity.com/paper/2103.15975/) (2021)
- [4-bit Quantization of LSTM-based Speech Recognition Models](https://mlsanity.com/paper/2108.12074/) (2021)
- [Cross-utterance Reranking Models with BERT and Graph Convolutional Networks for Conversational Speech Recognition](https://mlsanity.com/paper/2106.06922/) (2021)
- [Enhancing and Adversarial: Improve ASR with Speaker Labels](https://mlsanity.com/paper/2211.06369/) (2022)
- [Directed Speech Separation for Automatic Speech Recognition of Long Form Conversational Speech](https://mlsanity.com/paper/2112.05863/) (2021)
- [Multilingual Word Error Rate Estimation: e-WER3](https://mlsanity.com/paper/2304.00649/) (2023)
- [You don't understand me!: Comparing ASR results for L1 and L2 speakers of Swedish](https://mlsanity.com/paper/2405.13379/) (2024)
- [Toward Zero Oracle Word Error Rate on the Switchboard Benchmark](https://mlsanity.com/paper/2206.06192/) (2022)
- [Adversarial Attacks on Speech Recognition Systems for Mission-Critical Applications: A Survey](https://mlsanity.com/paper/2202.10594/) (2022)
- [JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching](https://mlsanity.com/paper/2506.23552/) (2025)
- [Automatic Speech Recognition in the Modern Era: Architectures, Training, and Evaluation](https://mlsanity.com/paper/2510.12827/) (2025)
- [A Benchmarking on Cloud based Speech-To-Text Services for French Speech and Background Noise Effect](https://mlsanity.com/paper/2105.03409/) (2021)
- [Reducing Exposure Bias in Training Recurrent Neural Network Transducers](https://mlsanity.com/paper/2108.10803/) (2021)
- [Asynchronous Decentralized Distributed Training of Acoustic Models](https://mlsanity.com/paper/2110.11199/) (2021)
- [VoiceMoji: A Novel On-Device Pipeline for Seamless Emoji Insertion in Dictation](https://mlsanity.com/paper/2112.12028/) (2021)
- [Soft Random Sampling: A Theoretical and Empirical Analysis](https://mlsanity.com/paper/2311.12727/) (2023)
- [Training Aware Sigmoidal Optimizer](https://mlsanity.com/paper/2102.08716/) (2021)
- [LT-LM: a novel non-autoregressive language model for single-shot lattice rescoring](https://mlsanity.com/paper/2104.02526/) (2021)

## Abstract (arXiv)

One of the most difficult speech recognition tasks is accurate recognition of human to human communication. Advances in deep learning over the last few years have produced major speech recognition improvements on the representative Switchboard conversational corpus. Word error rates that just a few years ago were 14% have dropped to 8.0%, then 6.6% and most recently 5.8%, and are now believed to be within striking range of human performance. This then raises two issues - what IS human performance, and how far down can we still drive speech recognition error rates? A recent paper by Microsoft suggests that we have already achieved human performance. In trying to verify this statement, we performed an independent set of human performance measurements on two conversational tasks and found that human performance may be considerably better than what was earlier reported, giving the community a significantly harder goal to achieve. We also report on our own efforts in this area, presenting a set of acoustic and language modeling techniques that lowered the word error rate of our own English conversational telephone LVCSR system to the level of 5.5%/10.3% on the Switchboard/CallHome subsets of the Hub5 2000 evaluation, which - at least at the writing of this paper - is a new performance milestone (albeit not at what we measure to be human performance!). On the acoustic side, we use a score fusion of three models: one LSTM with multiple feature inputs, a second LSTM trained with speaker-adversarial multi-task learning and a third residual net (ResNet) with 25 convolutional layers and time-dilated convolutions. On the language modeling side, we use word and character LSTMs and convolutional WaveNet-style language models.