# Self-Attention Transducers for End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/1909.13037 · HTML: https://mlsanity.com/paper/1909.13037/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2019.

## Built on by (most cited first, 24 of 24)

- [Efficient Sequence Transduction by Jointly Predicting Tokens and Durations](https://mlsanity.com/paper/2304.06795/) (2023)
- [Context-Aware Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2111.03250/) (2021)
- [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021)
- [Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition](https://mlsanity.com/paper/2109.09161/) (2021)
- [SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing](https://mlsanity.com/paper/2302.14638/) (2023)
- [Non-autoregressive Transformer-based End-to-end ASR using BERT](https://mlsanity.com/paper/2104.04805/) (2021)
- [FSR: Accelerating the Inference Process of Transducer-Based Models by Applying Fast-Skip Regularization](https://mlsanity.com/paper/2104.02882/) (2021)
- [Multi-Channel Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2108.12953/) (2021)
- [TSNAT: Two-Step Non-Autoregressvie Transformer Models for Speech Recognition](https://mlsanity.com/paper/2104.01522/) (2021)
- [Speech-to-SQL: Towards Speech-driven SQL Query Generation From Natural Language Question](https://mlsanity.com/paper/2201.01209/) (2022)
- [Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments](https://mlsanity.com/paper/2307.03354/) (2023)
- [Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition](https://mlsanity.com/paper/2102.07259/) (2021)
- [Transformer-based end-to-end speech recognition with residual Gaussian-based self-attention](https://mlsanity.com/paper/2103.15722/) (2021)
- [Conformer-based End-to-end Speech Recognition With Rotary Position Embedding](https://mlsanity.com/paper/2107.05907/) (2021)
- [On the Effectiveness of Pinyin-Character Dual-Decoding for End-to-End Mandarin Chinese ASR](https://mlsanity.com/paper/2201.10792/) (2022)
- [Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition](https://mlsanity.com/paper/2209.15176/) (2022)
- [Dual-Attention Neural Transducers for Efficient Wake Word Spotting in Speech Recognition](https://mlsanity.com/paper/2304.01905/) (2023)
- [Sim-T: Simplify the Transformer Network by Multiplexing Technique for Speech Recognition](https://mlsanity.com/paper/2304.04991/) (2023)
- [XCB: an effective contextual biasing approach to bias cross-lingual phrases in speech recognition](https://mlsanity.com/paper/2408.10524/) (2024)
- [Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducers](https://mlsanity.com/paper/2502.05232/) (2025)
- [A Lexical-aware Non-autoregressive Transformer-based ASR Model](https://mlsanity.com/paper/2305.10839/) (2023)
- [GNCformer Enhanced Self-attention for Automatic Speech Recognition](https://mlsanity.com/paper/2305.12755/) (2023)
- [Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis](https://mlsanity.com/paper/2310.05374/) (2023)
- [A Decade of Deep Learning: A Survey on The Magnificent Seven](https://mlsanity.com/paper/2412.16188/) (2024)

## Abstract (arXiv)

Recurrent neural network transducers (RNN-T) have been successfully applied in end-to-end speech recognition. However, the recurrent structure makes it difficult for parallelization . In this paper, we propose a self-attention transducer (SA-T) for speech recognition. RNNs are replaced with self-attention blocks, which are powerful to model long-term dependencies inside sequences and able to be efficiently parallelized. Furthermore, a path-aware regularization is proposed to assist SA-T to learn alignments and improve the performance. Additionally, a chunk-flow mechanism is utilized to achieve online decoding. All experiments are conducted on a Mandarin Chinese dataset AISHELL-1. The results demonstrate that our proposed approach achieves a 21.3% relative reduction in character error rate compared with the baseline RNN-T. In addition, the SA-T with chunk-flow mechanism can perform online decoding with only a little degradation of the performance.