# Super-Human Performance in Online Low-latency Recognition of Conversational Speech

arXiv: https://arxiv.org/abs/2010.03449 · HTML: https://mlsanity.com/paper/2010.03449/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2020.

## Built on by (most cited first, 20 of 20)

- [Robust Self-Supervised Audio-Visual Speech Recognition](https://mlsanity.com/paper/2201.01763/) (2022)
- [Sequence Length is a Domain: Length-based Overfitting in Transformer Models](https://mlsanity.com/paper/2109.07276/) (2021)
- [CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022](https://mlsanity.com/paper/2204.06028/) (2022)
- [Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of Videos](https://mlsanity.com/paper/2206.04523/) (2022)
- [Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition](https://mlsanity.com/paper/2103.00422/) (2021)
- [Lost in Interpreting: Speech Translation from Source or Interpreter?](https://mlsanity.com/paper/2106.09343/) (2021)
- [Simultaneous Machine Translation with Large Language Models](https://mlsanity.com/paper/2309.06706/) (2023)
- [Cocktail-Party Audio-Visual Speech Recognition](https://mlsanity.com/paper/2506.02178/) (2025)
- [Toward Zero Oracle Word Error Rate on the Switchboard Benchmark](https://mlsanity.com/paper/2206.06192/) (2022)
- [Conversational SimulMT: Efficient Simultaneous Translation with Large Language Models](https://mlsanity.com/paper/2402.10552/) (2024)
- [A Cocktail-Party Benchmark: Multi-Modal dataset and Comparative Evaluation Results](https://mlsanity.com/paper/2510.23276/) (2025)
- [Towards efficient end-to-end speech recognition with biologically-inspired neural networks](https://mlsanity.com/paper/2110.02743/) (2021)
- [Robustness of Multi-Source MT to Transcription Errors](https://mlsanity.com/paper/2305.16894/) (2023)
- [Convoifilter: A case study of doing cocktail party speech recognition](https://mlsanity.com/paper/2308.11380/) (2023)
- [Practice of the conformer enhanced AUDIO-VISUAL HUBERT on Mandarin and English](https://mlsanity.com/paper/2303.12187/) (2023)
- [R-BI: Regularized Batched Inputs enhance Incremental Decoding Framework for Low-Latency Simultaneous Speech Translation](https://mlsanity.com/paper/2401.05700/) (2024)
- [The AI Co-Ethnographer: How Far Can Automation Take Qualitative Research?](https://mlsanity.com/paper/2505.00012/) (2025)
- [Streaming Non-Autoregressive Model for Accent Conversion and Pronunciation Improvement](https://mlsanity.com/paper/2506.16580/) (2025)
- [Short-Term Word-Learning in a Dynamically Changing Environment](https://mlsanity.com/paper/2203.15404/) (2022)
- [SOI: Scaling Down Computational Complexity by Estimating Partial States of the Model](https://mlsanity.com/paper/2410.03813/) (2024)

## Abstract (arXiv)

Achieving super-human performance in recognizing human speech has been a goal for several decades, as researchers have worked on increasingly challenging tasks. In the 1990's it was discovered, that conversational speech between two humans turns out to be considerably more difficult than read speech as hesitations, disfluencies, false starts and sloppy articulation complicate acoustic processing and require robust handling of acoustic, lexical and language context, jointly. Early attempts with statistical models could only reach error rates over 50% and far from human performance (WER of around 5.5%). Neural hybrid models and recent attention-based encoder-decoder models have considerably improved performance as such contexts can now be learned in an integral fashion. However, processing such contexts requires an entire utterance presentation and thus introduces unwanted delays before a recognition result can be output. In this paper, we address performance as well as latency. We present results for a system that can achieve super-human performance (at a WER of 5.0%, over the Switchboard conversational benchmark) at a word based latency of only 1 second behind a speaker's speech. The system uses multiple attention-based encoder-decoder networks integrated within a novel low latency incremental inference approach.