# Transformer ASR with Contextual Block Processing

arXiv: https://arxiv.org/abs/1910.07204 · HTML: https://mlsanity.com/paper/1910.07204/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2019.

## Built on by (most cited first, 28 of 28)

- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Variable Attention Masking for Configurable Transformer Transducer Speech Recognition](https://mlsanity.com/paper/2211.01438/) (2022)
- [Decoder-only Architecture for Streaming End-to-end Speech Recognition](https://mlsanity.com/paper/2406.16107/) (2024)
- [CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASR](https://mlsanity.com/paper/2203.16758/) (2022)
- [Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation](https://mlsanity.com/paper/2204.08920/) (2022)
- [BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model](https://mlsanity.com/paper/2210.16663/) (2022)
- [Wake Word Detection with Streaming Transformers](https://mlsanity.com/paper/2102.04488/) (2021)
- [Streaming Transformer for Hardware Efficient Voice Trigger Detection and False Trigger Mitigation](https://mlsanity.com/paper/2105.06598/) (2021)
- [Contextualized Automatic Speech Recognition with Dynamic Vocabulary](https://mlsanity.com/paper/2405.13344/) (2024)
- [Mamba for Streaming ASR Combined with Unimodal Aggregation](https://mlsanity.com/paper/2410.00070/) (2024)
- [Context-aware Fine-tuning of Self-supervised Speech Models](https://mlsanity.com/paper/2212.08542/) (2022)
- [Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognition](https://mlsanity.com/paper/2309.08436/) (2023)
- [Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2104.00120/) (2021)
- [Streaming Joint Speech Recognition and Disfluency Detection](https://mlsanity.com/paper/2211.08726/) (2022)
- [Towards Effective and Compact Contextual Representation for Conformer Transducer Speech Recognition Systems](https://mlsanity.com/paper/2306.13307/) (2023)
- [Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments](https://mlsanity.com/paper/2307.03354/) (2023)
- [Exploring SSL Discrete Tokens for Multilingual ASR](https://mlsanity.com/paper/2409.08805/) (2024)
- [Semi-Autoregressive Streaming ASR With Label Context](https://mlsanity.com/paper/2309.10926/) (2023)
- [Streaming on-device detection of device directed speech from voice and touch-based invocation](https://mlsanity.com/paper/2110.04656/) (2021)
- [Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models](https://mlsanity.com/paper/2104.14346/) (2021)
- [Chunk Based Speech Pre-training with High Resolution Finite Scalar Quantization](https://mlsanity.com/paper/2509.15579/) (2025)
- [DCTX-Conformer: Dynamic context carry-over for low latency unified streaming and non-streaming Conformer ASR](https://mlsanity.com/paper/2306.08175/) (2023)
- [Memory-augmented conformer for improved end-to-end long-form ASR](https://mlsanity.com/paper/2309.13029/) (2023)
- [The evaluation of a code-switched Sepedi-English automatic speech recognition system](https://mlsanity.com/paper/2403.07947/) (2024)
- [Joint Optimization of Streaming and Non-Streaming Automatic Speech Recognition with Multi-Decoder and Knowledge Distillation](https://mlsanity.com/paper/2405.13514/) (2024)
- [Spiralformer: Low Latency Encoder for Streaming Speech Recognition with Circular Layer Skipping and Early Exiting](https://mlsanity.com/paper/2510.00982/) (2025)

## Abstract (arXiv)

The Transformer self-attention network has recently shown promising performance as an alternative to recurrent neural networks (RNNs) in end-to-end (E2E) automatic speech recognition (ASR) systems. However, the Transformer has a drawback in that the entire input sequence is required to compute self-attention. In this paper, we propose a new block processing method for the Transformer encoder by introducing a context-aware inheritance mechanism. An additional context embedding vector handed over from the previously processed block helps to encode not only local acoustic information but also global linguistic, channel, and speaker attributes. We introduce a novel mask technique to implement the context inheritance to train the model efficiently. Evaluations of the Wall Street Journal (WSJ), Librispeech, VoxForge Italian, and AISHELL-1 Mandarin speech recognition datasets show that our proposed contextual block processing method outperforms naive block processing consistently. Furthermore, the attention weight tendency of each layer is analyzed to clarify how the added contextual inheritance mechanism models the global information.