# Self-Attention Aligner: A Latency-Control End-to-End Model for ASR Using Self-Attention Network and Chunk-Hopping

arXiv: https://arxiv.org/abs/1902.06450 · HTML: https://mlsanity.com/paper/1902.06450/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2019.

## Built on by (most cited first, 30 of 33)

- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Dual Causal/Non-Causal Self-Attention for Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2107.01269/) (2021)
- [The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021](https://mlsanity.com/paper/2107.00279/) (2021)
- [ZeroPrompt: Streaming Acoustic Encoders are Zero-Shot Masked LMs](https://mlsanity.com/paper/2305.10649/) (2023)
- [Decoder-only Architecture for Streaming End-to-end Speech Recognition](https://mlsanity.com/paper/2406.16107/) (2024)
- [FSR: Accelerating the Inference Process of Transducer-Based Models by Applying Fast-Skip Regularization](https://mlsanity.com/paper/2104.02882/) (2021)
- [CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASR](https://mlsanity.com/paper/2203.16758/) (2022)
- [Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation](https://mlsanity.com/paper/2301.13003/) (2023)
- [Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latency](https://mlsanity.com/paper/2104.02176/) (2021)
- [Folding Attention: Memory and Power Optimization for On-Device Transformer-based Streaming Speech Recognition](https://mlsanity.com/paper/2309.07988/) (2023)
- [Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognition](https://mlsanity.com/paper/2309.08436/) (2023)
- [Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition](https://mlsanity.com/paper/2102.07259/) (2021)
- [Semi-Autoregressive Streaming ASR With Label Context](https://mlsanity.com/paper/2309.10926/) (2023)
- [Shiftable Context: Addressing Training-Inference Context Mismatch in Simultaneous Speech Translation](https://mlsanity.com/paper/2307.01377/) (2023)
- [Implicit Memory Transformer for Computationally Efficient Simultaneous Speech Translation](https://mlsanity.com/paper/2307.01381/) (2023)
- [Flexi-Transducer: Optimizing Latency, Accuracy and Compute forMulti-Domain On-Device Scenarios](https://mlsanity.com/paper/2104.02232/) (2021)
- [CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training](https://mlsanity.com/paper/2305.17499/) (2023)
- [Improved Training for End-to-End Streaming Automatic Speech Recognition Model with Punctuation](https://mlsanity.com/paper/2306.01296/) (2023)
- [Deformable TDNN with adaptive receptive fields for speech recognition](https://mlsanity.com/paper/2104.14791/) (2021)
- [Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition](https://mlsanity.com/paper/2209.15176/) (2022)
- [Sim-T: Simplify the Transformer Network by Multiplexing Technique for Speech Recognition](https://mlsanity.com/paper/2304.04991/) (2023)
- [Bigger is not Always Better: The Effect of Context Size on Speech Pre-Training](https://mlsanity.com/paper/2312.01515/) (2023)
- [Incorporating VAD into ASR System by Multi-task Learning](https://mlsanity.com/paper/2103.01661/) (2021)
- [StableEmit: Selection Probability Discount for Reducing Emission Latency of Streaming Monotonic Attention ASR](https://mlsanity.com/paper/2107.00635/) (2021)
- [VAD-free Streaming Hybrid CTC/Attention ASR for Unsegmented Recording](https://mlsanity.com/paper/2107.07509/) (2021)
- [StreaMulT: Streaming Multimodal Transformer for Heterogeneous and Arbitrary Long Sequential Data](https://mlsanity.com/paper/2110.08021/) (2021)
- [Mutually-Constrained Monotonic Multihead Attention for Online ASR](https://mlsanity.com/paper/2103.14302/) (2021)
- [GNCformer Enhanced Self-attention for Automatic Speech Recognition](https://mlsanity.com/paper/2305.12755/) (2023)
- [Enhancing the Unified Streaming and Non-streaming Model with Contrastive Learning](https://mlsanity.com/paper/2306.00755/) (2023)

## Abstract (arXiv)

Self-attention network, an attention-based feedforward neural network, has recently shown the potential to replace recurrent neural networks (RNNs) in a variety of NLP tasks. However, it is not clear if the self-attention network could be a good alternative of RNNs in automatic speech recognition (ASR), which processes the longer speech sequences and may have online recognition requirements. In this paper, we present a RNN-free end-to-end model: self-attention aligner (SAA), which applies the self-attention networks to a simplified recurrent neural aligner (RNA) framework. We also propose a chunk-hopping mechanism, which enables the SAA model to encode on segmented frame chunks one after another to support online recognition. Experiments on two Mandarin ASR datasets show the replacement of RNNs by the self-attention networks yields a 8.4%-10.2% relative character error rate (CER) reduction. In addition, the chunk-hopping mechanism allows the SAA to have only a 2.5% relative CER degradation with a 320ms latency. After jointly training with a self-attention network language model, our SAA model obtains further error rate reduction on multiple datasets. Especially, it achieves 24.12% CER on the Mandarin ASR benchmark (HKUST), exceeding the best end-to-end model by over 2% absolute CER.