# On Comparison of Encoders for Attention based End to End Speech Recognition in Standalone and Rescoring Mode

arXiv: https://arxiv.org/abs/2206.12829 · HTML: https://mlsanity.com/paper/2206.12829/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 26 Jun 2022.

## Builds on (most cited first, 21 of 21)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [Deliberation Model Based Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/2003.07962/) (2020)
- [Correction of Automatic Speech Recognition with Transformer Sequence-to-sequence Model](https://mlsanity.com/paper/1910.10697/) (2019)
- [Towards Fast and Accurate Streaming End-to-End ASR](https://mlsanity.com/paper/2004.11544/) (2020)
- [Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM](https://mlsanity.com/paper/1706.02737/) (2017)
- [RescoreBERT: Discriminative Speech Recognition Rescoring with BERT](https://mlsanity.com/paper/2202.01094/) (2022)
- [Transformer Based Deliberation for Two-Pass Speech Recognition](https://mlsanity.com/paper/2101.11577/) (2021)
- [Parallel Rescoring with Transformer for Streaming On-Device Speech Recognition](https://mlsanity.com/paper/2008.13093/) (2020)
- [A Simple Baseline for Domain Adaptation in End to End ASR Systems Using Synthetic Data](https://mlsanity.com/paper/2206.13240/) (2022)
- [Attention based end to end Speech Recognition for Voice Search in Hindi and English](https://mlsanity.com/paper/2111.10208/) (2021)

## Built on by (most cited first, 2 of 2)

- [Rapid Speaker Adaptation in Low Resource Text to Speech Systems using Synthetic Data and Transfer learning](https://mlsanity.com/paper/2312.01107/) (2023)
- [Code-Mixed Text to Speech Synthesis under Low-Resource Constraints](https://mlsanity.com/paper/2312.01103/) (2023)

## Abstract (arXiv)

The streaming automatic speech recognition (ASR) models are more popular and suitable for voice-based applications. However, non-streaming models provide better performance as they look at the entire audio context. To leverage the benefits of the non-streaming model in streaming applications like voice search, it is commonly used in second pass re-scoring mode. The candidate hypothesis generated using steaming models is re-scored using a non-streaming model. In this work, we evaluate the non-streaming attention-based end-to-end ASR models on the Flipkart voice search task in both standalone and re-scoring modes. These models are based on Listen-Attend-Spell (LAS) encoder-decoder architecture. We experiment with different encoder variations based on LSTM, Transformer, and Conformer. We compare the latency requirements of these models along with their performance. Overall we show that the Transformer model offers acceptable WER with the lowest latency requirements. We report a relative WER improvement of around 16% with the second pass LAS re-scoring with latency overhead under 5ms. We also highlight the importance of CNN front-end with Transformer architecture to achieve comparable word error rates (WER). Moreover, we observe that in the second pass re-scoring mode all the encoders provide similar benefits whereas the difference in performance is prominent in standalone text generation mode.