# Deliberation Model Based Two-Pass End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/2003.07962 · HTML: https://mlsanity.com/paper/2003.07962/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Mar 2020.

## Built on by (most cited first, 30 of 57)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://mlsanity.com/paper/2212.08055/) (2022)
- [Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition](https://mlsanity.com/paper/2310.06434/) (2023)
- [WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2203.15455/) (2022)
- [FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition](https://mlsanity.com/paper/2105.03842/) (2021)
- [HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models](https://mlsanity.com/paper/2309.15701/) (2023)
- [RescoreBERT: Discriminative Speech Recognition Rescoring with BERT](https://mlsanity.com/paper/2202.01094/) (2022)
- [Large Language Models are Efficient Learners of Noise-Robust Speech Recognition](https://mlsanity.com/paper/2401.10446/) (2024)
- [Instant One-Shot Word-Learning for Context-Specific Neural Sequence-to-Sequence Speech Recognition](https://mlsanity.com/paper/2107.02268/) (2021)
- [Transformer Based Deliberation for Two-Pass Speech Recognition](https://mlsanity.com/paper/2101.11577/) (2021)
- [Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks](https://mlsanity.com/paper/2105.00573/) (2021)
- [Learning Word-Level Confidence For Subword End-to-End ASR](https://mlsanity.com/paper/2103.06716/) (2021)
- [It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition](https://mlsanity.com/paper/2402.05457/) (2024)
- [A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding](https://mlsanity.com/paper/2211.05869/) (2022)
- [Towards Contextual Spelling Correction for Customization of End-to-end Speech Recognition Systems](https://mlsanity.com/paper/2203.00888/) (2022)
- [Chain-of-Thought Prompting for Speech Translation](https://mlsanity.com/paper/2409.11538/) (2024)
- [Deliberation of Streaming RNN-Transducer by Non-autoregressive Decoding](https://mlsanity.com/paper/2112.11442/) (2021)
- [Improving Deliberation by Text-Only and Semi-Supervised Training](https://mlsanity.com/paper/2206.14716/) (2022)
- [SoftCorrect: Error Correction with Soft Detection for Automatic Speech Recognition](https://mlsanity.com/paper/2212.01039/) (2022)
- [Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentation](https://mlsanity.com/paper/2302.11192/) (2023)
- [Streaming parallel transducer beam search with fast-slow cascaded encoders](https://mlsanity.com/paper/2203.15773/) (2022)
- [Deliberation Model for On-Device Spoken Language Understanding](https://mlsanity.com/paper/2204.01893/) (2022)
- [Two-Pass Low Latency End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2207.06670/) (2022)
- [Contextualized Automatic Speech Recognition with Dynamic Vocabulary](https://mlsanity.com/paper/2405.13344/) (2024)
- [TSNAT: Two-Step Non-Autoregressvie Transformer Models for Speech Recognition](https://mlsanity.com/paper/2104.01522/) (2021)
- [Have best of both worlds: two-pass hybrid and E2E cascading framework for speech recognition](https://mlsanity.com/paper/2110.04891/) (2021)
- [KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI](https://mlsanity.com/paper/2510.02327/) (2025)
- [Lattention: Lattice-attention in ASR rescoring](https://mlsanity.com/paper/2111.10157/) (2021)

## Abstract (arXiv)

End-to-end (E2E) models have made rapid progress in automatic speech recognition (ASR) and perform competitively relative to conventional models. To further improve the quality, a two-pass model has been proposed to rescore streamed hypotheses using the non-streaming Listen, Attend and Spell (LAS) model while maintaining a reasonable latency. The model attends to acoustics to rescore hypotheses, as opposed to a class of neural correction models that use only first-pass text hypotheses. In this work, we propose to attend to both acoustics and first-pass hypotheses using a deliberation network. A bidirectional encoder is used to extract context information from first-pass hypotheses. The proposed deliberation model achieves 12% relative WER reduction compared to LAS rescoring in Google Voice Search (VS) tasks, and 23% reduction on a proper noun test set. Compared to a large conventional model, our best model performs 21% relatively better for VS. In terms of computational complexity, the deliberation decoder has a larger size than the LAS decoder, and hence requires more computations in second-pass decoding.