# Transformer-based Online CTC/attention End-to-End Speech Recognition Architecture

arXiv: https://arxiv.org/abs/2001.08290 · HTML: https://mlsanity.com/paper/2001.08290/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jan 2020.

## Built on by (most cited first, 28 of 28)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2203.15455/) (2022)
- [Parallelizing Legendre Memory Unit Training](https://mlsanity.com/paper/2102.11417/) (2021)
- [Dual Causal/Non-Causal Self-Attention for Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2107.01269/) (2021)
- [CopyNE: Better Contextual ASR by Copying Named Entities](https://mlsanity.com/paper/2305.12839/) (2023)
- [Towards Online Continuous Sign Language Recognition and Translation](https://mlsanity.com/paper/2401.05336/) (2024)
- [Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models](https://mlsanity.com/paper/2107.09428/) (2021)
- [CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASR](https://mlsanity.com/paper/2203.16758/) (2022)
- [WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2104.03587/) (2021)
- [Boosting Cross-Domain Speech Recognition with Self-Supervision](https://mlsanity.com/paper/2206.09783/) (2022)
- [A Melody-Unsupervision Model for Singing Voice Synthesis](https://mlsanity.com/paper/2110.06546/) (2021)
- [EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wild](https://mlsanity.com/paper/2502.14892/) (2025)
- [Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition](https://mlsanity.com/paper/2102.07259/) (2021)
- [Three-Module Modeling For End-to-End Spoken Language Understanding Using Pre-trained DNN-HMM-Based Acoustic-Phonetic Model](https://mlsanity.com/paper/2204.03315/) (2022)
- [Improved Training for End-to-End Streaming Automatic Speech Recognition Model with Punctuation](https://mlsanity.com/paper/2306.01296/) (2023)
- [Deformable TDNN with adaptive receptive fields for speech recognition](https://mlsanity.com/paper/2104.14791/) (2021)
- [Language-Universal Adapter Learning with Knowledge Distillation for End-to-End Multilingual Speech Recognition](https://mlsanity.com/paper/2303.01249/) (2023)
- [Multi-Variant Consistency based Self-supervised Learning for Robust Automatic Speech Recognition](https://mlsanity.com/paper/2112.12522/) (2021)
- [Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognition](https://mlsanity.com/paper/2302.09723/) (2023)
- [Mutually-Constrained Monotonic Multihead Attention for Online ASR](https://mlsanity.com/paper/2103.14302/) (2021)
- [Deep Sparse Conformer for Speech Recognition](https://mlsanity.com/paper/2209.00260/) (2022)
- [Attention Enhanced Citrinet for Speech Recognition](https://mlsanity.com/paper/2209.00261/) (2022)
- [Streaming Speech-to-Confusion Network Speech Recognition](https://mlsanity.com/paper/2306.03778/) (2023)
- [DCTX-Conformer: Dynamic context carry-over for low latency unified streaming and non-streaming Conformer ASR](https://mlsanity.com/paper/2306.08175/) (2023)
- [Research on an improved Conformer end-to-end Speech Recognition Model with R-Drop Structure](https://mlsanity.com/paper/2306.08329/) (2023)
- [A Decade of Deep Learning: A Survey on The Magnificent Seven](https://mlsanity.com/paper/2412.16188/) (2024)
- [Spiralformer: Low Latency Encoder for Streaming Speech Recognition with Circular Layer Skipping and Early Exiting](https://mlsanity.com/paper/2510.00982/) (2025)

## Abstract (arXiv)

Recently, Transformer has gained success in automatic speech recognition (ASR) field. However, it is challenging to deploy a Transformer-based end-to-end (E2E) model for online speech recognition. In this paper, we propose the Transformer-based online CTC/attention E2E ASR architecture, which contains the chunk self-attention encoder (chunk-SAE) and the monotonic truncated attention (MTA) based self-attention decoder (SAD). Firstly, the chunk-SAE splits the speech into isolated chunks. To reduce the computational cost and improve the performance, we propose the state reuse chunk-SAE. Sencondly, the MTA based SAD truncates the speech features monotonically and performs attention on the truncated features. To support the online recognition, we integrate the state reuse chunk-SAE and the MTA based SAD into online CTC/attention architecture. We evaluate the proposed online models on the HKUST Mandarin ASR benchmark and achieve a 23.66% character error rate (CER) with a 320 ms latency. Our online model yields as little as 0.19% absolute CER degradation compared with the offline baseline, and achieves significant improvement over our prior work on Long Short-Term Memory (LSTM) based online E2E models.