mlsanityData to 7 Oct 2026

Monotonic Chunkwise Attention

arXiv 1712.05382 ↗First posted 14 Dec 2017cs.CL stat.MLcited by 82

Not on the map. Why?

Abstract (arXiv)

Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction. To address these issues, we propose Monotonic Chunkwise Attention (MoChA), which adaptively splits the input sequence into small chunks over which soft attention is computed. We show that models utilizing MoChA can be trained efficiently with standard backpropagation while allowing online and linear-time decoding at test time. When applied to online speech recognition, we obtain state-of-the-art results and match the performance of a model using an offline soft attention mechanism. In document summarization experiments where we do not expect monotonic alignments, we show significantly improved performance compared to a baseline monotonic attention-based model.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 19; 2023: 18; 2024: 10; 2025: 5; 2026: 5.

82 papers.

Sort:

  1. 2021A Survey on Neural Speech Synthesiscited by 182
  2. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  3. 2023End-to-End Speech Recognition: A Surveycited by 127
  4. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 123
  5. 2021Translating Images into Mapscited by 115
  6. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 37
  7. 2024TransLLaMa: LLM-based Simultaneous Translation Systemcited by 34
  8. 2021Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERTcited by 31
  9. 2021Factorized Neural Transducer for Efficient Language Model Adaptationcited by 31
  10. 2022Large-Scale Streaming End-to-End Speech Translation with Neural Transducerscited by 31
  11. 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 24
  12. 2023A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speechcited by 22
38 more
  1. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
  2. 2022Accelerating RNN-T Training and Inference Using CTC guidancecited by 17
  3. 2022LSG Attention: Extrapolation of pretrained Transformers to long sequencescited by 16
  4. 2021Dual Causal/Non-Causal Self-Attention for Streaming End-to-End Speech Recognitioncited by 14
  5. 2024Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Timecited by 14
  6. 2022A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capabilitycited by 12
  7. 2021Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognitioncited by 11
  8. 2021Streaming End-to-End ASR based on Blockwise Non-Autoregressive Modelscited by 11
  9. 2022CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASRcited by 11
  10. 2022Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translationcited by 11
  11. 2021WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognitioncited by 10
  12. 2024A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translationcited by 10
  13. 2024Efficient Streaming LLM for Speech Recognitioncited by 10
  14. 2021Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attentioncited by 9
  15. 2023Simultaneous Machine Translation with Large Language Modelscited by 9
  16. 2021Multi-Channel Transformer Transducer for Speech Recognitioncited by 8
  17. 2022ConvRNN-T: Convolutional Augmented Recurrent Neural Network Transducers for Streaming Speech Recognitioncited by 8
  18. 2022Minimum Latency Training of Sequence Transducers for Streaming End-to-End Speech Recognitioncited by 8
  19. 2023Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoffcited by 8
  20. 2023Efficient Monotonic Multihead Attentioncited by 8
  21. 2021Language model fusion for streaming end to end speech recognitioncited by 7
  22. 2023Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognitioncited by 7
  23. 2021Multi-mode Transformer Transducer with Stochastic Future Contextcited by 6
  24. 2021Decision Attentive Regularization to Improve Simultaneous Speech Translation Systemscited by 6
  25. 2022Streaming Joint Speech Recognition and Disfluency Detectioncited by 6
  26. 2023Unified Segment-to-Segment Framework for Simultaneous Sequence Generationcited by 6
  27. 2026FBS: Modeling Native Parallel Reading inside a Transformercited by 6
  28. 2021Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognitioncited by 5
  29. 2021A study of latent monotonic attention variantscited by 5
  30. 2021On Biasing Transformer Attention Towards Monotonicitycited by 5
  31. 2022Anticipation-Free Training for Simultaneous Machine Translationcited by 5
  32. 2023TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Modelcited by 5
  33. 2021Streaming end-to-end speech recognition with jointly trained neural feature enhancementcited by 4
  34. 2022Monotonic segmental attention for automatic speech recognitioncited by 4
  35. 2023Efficient Encoders for Streaming Sequence Taggingcited by 4
  36. 2024Overcoming Non-monotonicity in Transducer-based Streaming Generationcited by 4
  37. 2022Integrating Lattice-Free MMI into End-to-End Speech Recognitioncited by 3
  38. 2022Memory-Efficient Training of RNN-Transducer with Sampled Softmaxcited by 3

50 most cited of 82 All 82 (JSON)