mlsanityData to 7 Oct 2026

A study of latent monotonic attention variants

arXiv 2103.16710 ↗First posted 30 Mar 2021cs.CL cs.AI cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

End-to-end models reach state-of-the-art performance for speech recognition, but global soft attention is not monotonic, which might lead to convergence problems, to instability, to bad generalisation, cannot be used for online streaming, and is also inefficient in calculation. Monotonicity can potentially fix all of this. There are several ad-hoc solutions or heuristics to introduce monotonicity, but a principled introduction is rarely found in literature so far. In this paper, we present a mathematically clean solution to introduce monotonicity, by introducing a new latent variable which represents the audio position or segment boundaries. We compare several monotonic latent models to our global soft attention baseline such as a hard attention model, a local windowed soft attention model, and a segmental soft attention model. We can show that our monotonic models perform as good as the global soft attention model. We perform our experiments on Switchboard 300h. We carefully outline the details of our training and release our code and configs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 35 of its 71 references that are in the map. First posted on arXiv 30 Mar 2021.

Builds on

35 of 71 references.

References · 35

  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  5. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  6. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  7. 2015Pointer Networkscited by 882
  8. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  9. 2014Recurrent Models of Visual Attentioncited by 344
  10. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  11. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  12. 2016Modeling Coverage for Neural Machine Translationcited by 110
23 more
  1. 2019Monotonic Infinite Lookback Attention for Simultaneous Machine Translationcited by 103
  2. 2017Online and Linear-Time Attention by Enforcing Monotonic Alignmentscited by 85
  3. 2017Monotonic Chunkwise Attentioncited by 82
  4. 2017Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Modelscited by 68
  5. 2019Two-Pass End-to-End Speech Recognitioncited by 62
  6. 2017Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attentioncited by 57
  7. 2017Exploring Neural Transducers for End-to-End Speech Recognitioncited by 43
  8. 2018Latent Alignment and Variational Attentioncited by 36
  9. 2020Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboardcited by 36
  10. 2018Improved training of end-to-end attention models for speech recognitioncited by 35
  11. 2020A New Training Pipeline for an Improved Neural Transducercited by 34
  12. 2019Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTScited by 33
  13. 2018RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognitioncited by 30
  14. 2017Variational Attention for Sequence-to-Sequence Modelscited by 29
  15. 2016The Neural Noisy Channelcited by 19
  16. 2018Forward Attention in Sequence-to-sequence Acoustic Modelling for Speech Synthesiscited by 17
  17. 2018On The Alignment Problem In Multi-Head Attention-Based Neural Machine Translationcited by 17
  18. 2016Online Segment to Segment Neural Transductioncited by 14
  19. 2018An Online Attention-based Model for Speech Recognitioncited by 13
  20. 2018Hard Non-Monotonic Attention for Character-Level Transductioncited by 12
  21. 2016Segmental Recurrent Neural Networks for End-to-end Speech Recognitioncited by 7
  22. 2017Learning Hard Alignments with Variational Inferencecited by 5
  23. 2018Extending Recurrent Neural Aligner for Streaming End-to-End Speech Recognition in Mandarincited by 5

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 1; 2023: 3.

5 papers.

  1. 2023End-to-End Speech Recognition: A Surveycited by 127
  2. 2023A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speechcited by 22
  3. 2023Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognitioncited by 7
  4. 2021Equivalence of Segmental and Neural Transducer Modeling: A Proof of Conceptcited by 5
  5. 2022Monotonic segmental attention for automatic speech recognitioncited by 4