# Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/2008.05750 · HTML: https://mlsanity.com/paper/2008.05750/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2020.

## Built on by (most cited first, 23 of 23)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer](https://mlsanity.com/paper/2106.04833/) (2021)
- [SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Training](https://mlsanity.com/paper/2201.10207/) (2022)
- [Accelerating Transducers through Adjacent Token Merging](https://mlsanity.com/paper/2306.16009/) (2023)
- [Variable Attention Masking for Configurable Transformer Transducer Speech Recognition](https://mlsanity.com/paper/2211.01438/) (2022)
- [Multi-Channel Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2108.12953/) (2021)
- [ConvRNN-T: Convolutional Augmented Recurrent Neural Network Transducers for Streaming Speech Recognition](https://mlsanity.com/paper/2209.14868/) (2022)
- [Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments](https://mlsanity.com/paper/2307.03354/) (2023)
- [Uconv-Conformer: High Reduction of Input Sequence Length for End-to-End Speech Recognition](https://mlsanity.com/paper/2208.07657/) (2022)
- [Replay to Remember: Continual Layer-Specific Fine-tuning for German Speech Recognition](https://mlsanity.com/paper/2307.07280/) (2023)
- [AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation](https://mlsanity.com/paper/2212.08911/) (2022)
- [Dynamic Acoustic Unit Augmentation With BPE-Dropout for Low-Resource End-to-End Speech Recognition](https://mlsanity.com/paper/2103.07186/) (2021)
- [Compute Cost Amortized Transformer for Streaming ASR](https://mlsanity.com/paper/2207.02393/) (2022)
- [Bridging the Granularity Gap for Acoustic Modeling](https://mlsanity.com/paper/2305.17356/) (2023)
- [Multilingual Speech Translation with Unified Transformer: Huawei Noah's Ark Lab at IWSLT 2021](https://mlsanity.com/paper/2106.00197/) (2021)
- [XCB: an effective contextual biasing approach to bias cross-lingual phrases in speech recognition](https://mlsanity.com/paper/2408.10524/) (2024)
- [CoCA-MDD: A Coupled Cross-Attention based Framework for Streaming Mispronunciation Detection and Diagnosis](https://mlsanity.com/paper/2111.08191/) (2021)
- [WavFT: Acoustic model finetuning with labelled and unlabelled data](https://mlsanity.com/paper/2204.00348/) (2022)
- [DCTX-Conformer: Dynamic context carry-over for low latency unified streaming and non-streaming Conformer ASR](https://mlsanity.com/paper/2306.08175/) (2023)
- [Globally Normalising the Transducer for Streaming Speech Recognition](https://mlsanity.com/paper/2307.10975/) (2023)
- [Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis](https://mlsanity.com/paper/2310.05374/) (2023)
- [sTransformer: A Modular Approach for Extracting Inter-Sequential and Temporal Information for Time-Series Forecasting](https://mlsanity.com/paper/2408.09723/) (2024)

## Abstract (arXiv)

Transformer has achieved competitive performance against state-of-the-art end-to-end models in automatic speech recognition (ASR), and requires significantly less training time than RNN-based models. The original Transformer, with encoder-decoder architecture, is only suitable for offline ASR. It relies on an attention mechanism to learn alignments, and encodes input audio bidirectionally. The high computation cost of Transformer decoding also limits its use in production streaming systems. To make Transformer suitable for streaming ASR, we explore Transducer framework as a streamable way to learn alignments. For audio encoding, we apply unidirectional Transformer with interleaved convolution layers. The interleaved convolution layers are used for modeling future context which is important to performance. To reduce computation cost, we gradually downsample acoustic input, also with the interleaved convolution layers. Moreover, we limit the length of history context in self-attention to maintain constant computation cost for each decoding step. We show that this architecture, named Conv-Transformer Transducer, achieves competitive performance on LibriSpeech dataset (3.6% WER on test-clean) without external language models. The performance is comparable to previously published streamable Transformer Transducer and strong hybrid streaming ASR systems, and is achieved with smaller look-ahead window (140~ms), fewer parameters and lower frame rate.