# Streaming automatic speech recognition with the transformer model

arXiv: https://arxiv.org/abs/2001.02674 · HTML: https://mlsanity.com/paper/2001.02674/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2020.

## Built on by (most cited first, 30 of 73)

- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [A Survey on Time-Series Pre-Trained Models](https://mlsanity.com/paper/2305.10716/) (2023)
- [Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR](https://mlsanity.com/paper/2106.06636/) (2021)
- [Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers](https://mlsanity.com/paper/2104.09426/) (2021)
- [Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition](https://mlsanity.com/paper/2312.17279/) (2023)
- [SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and Effective Hotword Customization Ability](https://mlsanity.com/paper/2308.03266/) (2023)
- [How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?](https://mlsanity.com/paper/2412.18495/) (2024)
- [Dual Causal/Non-Causal Self-Attention for Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2107.01269/) (2021)
- [Variable Attention Masking for Configurable Transformer Transducer Speech Recognition](https://mlsanity.com/paper/2211.01438/) (2022)
- [Chronological Thinking in Full-Duplex Spoken Dialogue Language Models](https://mlsanity.com/paper/2510.05150/) (2025)
- [Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition](https://mlsanity.com/paper/2103.00422/) (2021)
- [Personal VAD 2.0: Optimizing Personal Voice Activity Detection for On-Device Speech Recognition](https://mlsanity.com/paper/2204.03793/) (2022)
- [Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models](https://mlsanity.com/paper/2107.09428/) (2021)
- [CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASR](https://mlsanity.com/paper/2203.16758/) (2022)
- [Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation](https://mlsanity.com/paper/2204.08920/) (2022)
- [Fake the Real: Backdoor Attack on Deep Speech Classification via Voice Conversion](https://mlsanity.com/paper/2306.15875/) (2023)
- [Solving Probability and Statistics Problems by Program Synthesis](https://mlsanity.com/paper/2111.08267/) (2021)
- [ConvRNN-T: Convolutional Augmented Recurrent Neural Network Transducers for Streaming Speech Recognition](https://mlsanity.com/paper/2209.14868/) (2022)
- [Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoff](https://mlsanity.com/paper/2309.11379/) (2023)
- [Mamba for Streaming ASR Combined with Unimodal Aggregation](https://mlsanity.com/paper/2410.00070/) (2024)
- [Folding Attention: Memory and Power Optimization for On-Device Transformer-based Streaming Speech Recognition](https://mlsanity.com/paper/2309.07988/) (2023)
- [SRU++: Pioneering Fast Recurrence with Attention for Speech Recognition](https://mlsanity.com/paper/2110.05571/) (2021)
- [Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments](https://mlsanity.com/paper/2307.03354/) (2023)
- [Unified Segment-to-Segment Framework for Simultaneous Sequence Generation](https://mlsanity.com/paper/2310.17940/) (2023)
- [Breaking Down Power Barriers in On-Device Streaming ASR: Insights and Solutions](https://mlsanity.com/paper/2402.13076/) (2024)
- [The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning](https://mlsanity.com/paper/2603.17837/) (2026)
- [Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition](https://mlsanity.com/paper/2102.07259/) (2021)

## Abstract (arXiv)

Encoder-decoder based sequence-to-sequence models have demonstrated state-of-the-art results in end-to-end automatic speech recognition (ASR). Recently, the transformer architecture, which uses self-attention to model temporal context information, has been shown to achieve significantly lower word error rates (WERs) compared to recurrent neural network (RNN) based system architectures. Despite its success, the practical usage is limited to offline ASR tasks, since encoder-decoder architectures typically require an entire speech utterance as input. In this work, we propose a transformer based end-to-end ASR system for streaming ASR, where an output must be generated shortly after each spoken word. To achieve this, we apply time-restricted self-attention for the encoder and triggered attention for the encoder-decoder attention mechanism. Our proposed streaming transformer architecture achieves 2.8% and 7.2% WER for the "clean" and "other" test data of LibriSpeech, which to our knowledge is the best published streaming end-to-end ASR result for this task.