# Synchronous Transformers for End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/1912.02958 · HTML: https://mlsanity.com/paper/1912.02958/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2019.

## Built on by (most cited first, 24 of 24)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure](https://mlsanity.com/paper/2101.00387/) (2021)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021)
- [U2++: Unified Two-pass Bidirectional End-to-end Model for Speech Recognition](https://mlsanity.com/paper/2106.05642/) (2021)
- [Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition](https://mlsanity.com/paper/2312.17279/) (2023)
- [Variable Attention Masking for Configurable Transformer Transducer Speech Recognition](https://mlsanity.com/paper/2211.01438/) (2022)
- [FSR: Accelerating the Inference Process of Transducer-Based Models by Applying Fast-Skip Regularization](https://mlsanity.com/paper/2104.02882/) (2021)
- [Wake Word Detection with Streaming Transformers](https://mlsanity.com/paper/2102.04488/) (2021)
- [Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognition](https://mlsanity.com/paper/2309.08436/) (2023)
- [Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detection](https://mlsanity.com/paper/2406.10052/) (2024)
- [Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments](https://mlsanity.com/paper/2307.03354/) (2023)
- [Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition](https://mlsanity.com/paper/2102.07259/) (2021)
- [Training-Free Acceleration of ViTs with Delayed Spatial Merging](https://mlsanity.com/paper/2303.02331/) (2023)
- [Shifted Chunk Encoder for Transformer Based Streaming End-to-End ASR](https://mlsanity.com/paper/2203.15206/) (2022)
- [Sim-T: Simplify the Transformer Network by Multiplexing Technique for Speech Recognition](https://mlsanity.com/paper/2304.04991/) (2023)
- [StreaMulT: Streaming Multimodal Transformer for Heterogeneous and Arbitrary Long Sequential Data](https://mlsanity.com/paper/2110.08021/) (2021)
- [SSCFormer: Push the Limit of Chunk-wise Conformer for Streaming ASR Using Sequentially Sampled Chunks and Chunked Causal Convolution](https://mlsanity.com/paper/2211.11419/) (2022)
- [MFLA: Monotonic Finite Look-ahead Attention for Streaming Speech Recognition](https://mlsanity.com/paper/2506.03722/) (2025)
- [FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognition](https://mlsanity.com/paper/2210.17079/) (2022)
- [GNCformer Enhanced Self-attention for Automatic Speech Recognition](https://mlsanity.com/paper/2305.12755/) (2023)
- [Streaming Speech-to-Confusion Network Speech Recognition](https://mlsanity.com/paper/2306.03778/) (2023)
- [Improving CTC-AED model with integrated-CTC and auxiliary loss regularization](https://mlsanity.com/paper/2308.08449/) (2023)

## Abstract (arXiv)

For most of the attention-based sequence-to-sequence models, the decoder predicts the output sequence conditioned on the entire input sequence processed by the encoder. The asynchronous problem between the encoding and decoding makes these models difficult to be applied for online speech recognition. In this paper, we propose a model named synchronous transformer to address this problem, which can predict the output sequence chunk by chunk. Once a fixed-length chunk of the input sequence is processed by the encoder, the decoder begins to predict symbols immediately. During training, a forward-backward algorithm is introduced to optimize all the possible alignment paths. Our model is evaluated on a Mandarin dataset AISHELL-1. The experiments show that the synchronous transformer is able to perform encoding and decoding synchronously, and achieves a character error rate of 8.91% on the test set.