# Attention based on-device streaming speech recognition with large speech corpus

arXiv: https://arxiv.org/abs/2001.00577 · HTML: https://mlsanity.com/paper/2001.00577/

## Key facts

- Cited by 17 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 2 Jan 2020.

## Built on by (most cited first, 17 of 17)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition](https://mlsanity.com/paper/2103.00422/) (2021)
- [WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2104.03587/) (2021)
- [Neural-FST Class Language Model for End-to-End Speech Recognition](https://mlsanity.com/paper/2201.11867/) (2022)
- [Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition](https://mlsanity.com/paper/2210.00169/) (2022)
- [Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition](https://mlsanity.com/paper/2102.07259/) (2021)
- [Streaming end-to-end speech recognition with jointly trained neural feature enhancement](https://mlsanity.com/paper/2105.01254/) (2021)
- [Relaxed Attention: A Simple Method to Boost Performance of End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2107.01275/) (2021)
- [DANCER: Entity Description Augmented Named Entity Corrector for Automatic Speech Recognition](https://mlsanity.com/paper/2403.17645/) (2024)
- [Flexi-Transducer: Optimizing Latency, Accuracy and Compute forMulti-Domain On-Device Scenarios](https://mlsanity.com/paper/2104.02232/) (2021)
- [Unified Modeling of Multi-Domain Multi-Device ASR Systems](https://mlsanity.com/paper/2205.06655/) (2022)
- [Noisy Training Improves E2E ASR for the Edge](https://mlsanity.com/paper/2107.04677/) (2021)
- [Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition](https://mlsanity.com/paper/2209.15176/) (2022)
- [An Effective Context-Balanced Adaptation Approach for Long-Tailed Speech Recognition](https://mlsanity.com/paper/2409.06468/) (2024)
- [Incorporating Class-based Language Model for Named Entity Recognition in Factorized Neural Transducer](https://mlsanity.com/paper/2309.07648/) (2023)

## Abstract (arXiv)

In this paper, we present a new on-device automatic speech recognition (ASR) system based on monotonic chunk-wise attention (MoChA) models trained with large (> 10K hours) corpus. We attained around 90% of a word recognition rate for general domain mainly by using joint training of connectionist temporal classifier (CTC) and cross entropy (CE) losses, minimum word error rate (MWER) training, layer-wise pre-training and data augmentation methods. In addition, we compressed our models by more than 3.4 times smaller using an iterative hyper low-rank approximation (LRA) method while minimizing the degradation in recognition accuracy. The memory footprint was further reduced with 8-bit quantization to bring down the final model size to lower than 39 MB. For on-demand adaptation, we fused the MoChA models with statistical n-gram models, and we could achieve a relatively 36% improvement on average in word error rate (WER) for target domains including the general domain.