# Monotonic Chunkwise Attention

arXiv: https://arxiv.org/abs/1712.05382 · HTML: https://mlsanity.com/paper/1712.05382/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2017.

## Built on by (most cited first, 30 of 83)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [Translating Images into Maps](https://mlsanity.com/paper/2110.00966/) (2021)
- [VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment](https://mlsanity.com/paper/2406.07855/) (2024)
- [TransLLaMa: LLM-based Simultaneous Translation System](https://mlsanity.com/paper/2402.04636/) (2024)
- [Factorized Neural Transducer for Efficient Language Model Adaptation](https://mlsanity.com/paper/2110.01500/) (2021)
- [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021)
- [Large-Scale Streaming End-to-End Speech Translation with Neural Transducers](https://mlsanity.com/paper/2204.05352/) (2022)
- [Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling](https://mlsanity.com/paper/2509.08753/) (2025)
- [A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech](https://mlsanity.com/paper/2302.04215/) (2023)
- [Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks](https://mlsanity.com/paper/2305.03101/) (2023)
- [Accelerating RNN-T Training and Inference Using CTC guidance](https://mlsanity.com/paper/2210.16481/) (2022)
- [LSG Attention: Extrapolation of pretrained Transformers to long sequences](https://mlsanity.com/paper/2210.15497/) (2022)
- [Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time](https://mlsanity.com/paper/2406.09569/) (2024)
- [Dual Causal/Non-Causal Self-Attention for Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2107.01269/) (2021)
- [Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition](https://mlsanity.com/paper/2103.00422/) (2021)
- [A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capability](https://mlsanity.com/paper/2211.02499/) (2022)
- [Efficient Streaming LLM for Speech Recognition](https://mlsanity.com/paper/2410.03752/) (2024)
- [Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models](https://mlsanity.com/paper/2107.09428/) (2021)
- [CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASR](https://mlsanity.com/paper/2203.16758/) (2022)
- [Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation](https://mlsanity.com/paper/2204.08920/) (2022)
- [WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2104.03587/) (2021)
- [A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2406.06937/) (2024)
- [Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attention](https://mlsanity.com/paper/2110.08250/) (2021)
- [Simultaneous Machine Translation with Large Language Models](https://mlsanity.com/paper/2309.06706/) (2023)
- [Language model fusion for streaming end to end speech recognition](https://mlsanity.com/paper/2104.04487/) (2021)
- [Multi-Channel Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2108.12953/) (2021)
- [ConvRNN-T: Convolutional Augmented Recurrent Neural Network Transducers for Streaming Speech Recognition](https://mlsanity.com/paper/2209.14868/) (2022)

## Abstract (arXiv)

Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction. To address these issues, we propose Monotonic Chunkwise Attention (MoChA), which adaptively splits the input sequence into small chunks over which soft attention is computed. We show that models utilizing MoChA can be trained efficiently with standard backpropagation while allowing online and linear-time decoding at test time. When applied to online speech recognition, we obtain state-of-the-art results and match the performance of a model using an offline soft attention mechanism. In document summarization experiments where we do not expect monotonic alignments, we show significantly improved performance compared to a baseline monotonic attention-based model.