# Dissecting User-Perceived Latency of On-Device E2E Speech Recognition

arXiv: https://arxiv.org/abs/2104.02207 · HTML: https://mlsanity.com/paper/2104.02207/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 6 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019)
- [Deep Shallow Fusion for RNN-T Personalization](https://mlsanity.com/paper/2011.07754/) (2020)
- [Towards Fast and Accurate Streaming End-to-End ASR](https://mlsanity.com/paper/2004.11544/) (2020)
- [Alignment Restricted Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2011.03072/) (2020)
- [Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memory](https://mlsanity.com/paper/2005.08042/) (2020)
- [Optimizing Speech Recognition For The Edge](https://mlsanity.com/paper/1909.12408/) (2019)
- [Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2010.13878/) (2020)
- [RNN-T For Latency Controlled ASR With Improved Beam Search](https://mlsanity.com/paper/1911.01629/) (2019)
- [Low Latency End-to-End Streaming Speech Recognition with a Scout Network](https://mlsanity.com/paper/2003.10369/) (2020)
- [Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASR](https://mlsanity.com/paper/2004.05009/) (2020)
- [Scaling Up Online Speech Recognition Using ConvNets](https://mlsanity.com/paper/2001.09727/) (2020)
- [On the efficient representation and execution of deep acoustic models](https://mlsanity.com/paper/1607.04683/) (2016)
- [High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model](https://mlsanity.com/paper/2003.07482/) (2020)
- [Analyzing the Quality and Stability of a Streaming End-to-End On-Device Speech Recognizer](https://mlsanity.com/paper/2006.01416/) (2020)

## Built on by (most cited first, 19 of 19)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [ZeroPrompt: Streaming Acoustic Encoders are Zero-Shot Masked LMs](https://mlsanity.com/paper/2305.10649/) (2023)
- [The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systems](https://mlsanity.com/paper/2307.15493/) (2023)
- [Minimum Latency Training of Sequence Transducers for Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2211.02333/) (2022)
- [Delay-penalized transducer for low-latency streaming ASR](https://mlsanity.com/paper/2211.00490/) (2022)
- [TrimTail: Low-Latency Streaming ASR with Simple but Effective Spectrogram-Level Length Penalty](https://mlsanity.com/paper/2211.00522/) (2022)
- [Personalized Predictive ASR for Latency Reduction in Voice Assistants](https://mlsanity.com/paper/2305.13794/) (2023)
- [Extreme Encoder Output Frame Rate Reduction: Improving Computational Latencies of Large End-to-End Models](https://mlsanity.com/paper/2402.17184/) (2024)
- [Unified End-to-End Speech Recognition and Endpointing for Fast and Efficient Speech Systems](https://mlsanity.com/paper/2211.00786/) (2022)
- [TODM: Train Once Deploy Many Efficient Supernet-Based RNN-T Compression For On-device ASR Models](https://mlsanity.com/paper/2309.01947/) (2023)
- [E2E Segmentation in a Two-Pass Cascaded Encoder ASR Model](https://mlsanity.com/paper/2211.15432/) (2022)
- [Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Model](https://mlsanity.com/paper/2309.13018/) (2023)
- [Semantic Segmentation with Bidirectional Language Models Improves Long-form ASR](https://mlsanity.com/paper/2305.18419/) (2023)
- [Assessing Latency in ASR Systems: A Methodological Perspective for Real-Time Use](https://mlsanity.com/paper/2409.05674/) (2024)
- [Delayed-KD: Delayed Knowledge Distillation based CTC for Low-Latency Streaming ASR](https://mlsanity.com/paper/2505.22069/) (2025)
- [Improving endpoint detection in end-to-end streaming ASR for conversational speech](https://mlsanity.com/paper/2505.17070/) (2025)
- [Spiralformer: Low Latency Encoder for Streaming Speech Recognition with Circular Layer Skipping and Early Exiting](https://mlsanity.com/paper/2510.00982/) (2025)
- [Building an ASR Solution for Training and Assessing Children's Reading](https://mlsanity.com/paper/2606.31508/) (2026)
- [Less can be More: What Aspects of Speech Drive End-of-Turn Detection](https://mlsanity.com/paper/2609.11066/) (2026)

## Abstract (arXiv)

As speech-enabled devices such as smartphones and smart speakers become increasingly ubiquitous, there is growing interest in building automatic speech recognition (ASR) systems that can run directly on-device; end-to-end (E2E) speech recognition models such as recurrent neural network transducers and their variants have recently emerged as prime candidates for this task. Apart from being accurate and compact, such systems need to decode speech with low user-perceived latency (UPL), producing words as soon as they are spoken. This work examines the impact of various techniques - model architectures, training criteria, decoding hyperparameters, and endpointer parameters - on UPL. Our analyses suggest that measures of model size (parameters, input chunk sizes), or measures of computation (e.g., FLOPS, RTF) that reflect the model's ability to process input frames are not always strongly correlated with observed UPL. Thus, conventional algorithmic latency measurements might be inadequate in accurately capturing latency observed when models are deployed on embedded devices. Instead, we find that factors affecting token emission latency, and endpointing behavior have a larger impact on UPL. We achieve the best trade-off between latency and word error rate when performing ASR jointly with endpointing, while utilizing the recently proposed alignment regularization mechanism.