mlsanityData to 14 Oct 2026

Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition

arXiv 2102.07259 ↗First posted 14 Feb 2021cs.SD cs.CL eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

Attention is a very popular and effective mechanism in artificial neural network-based sequence-to-sequence models. In this survey paper, a comprehensive review of the different attention models used in developing automatic speech recognition systems is provided. The paper focuses on the development and evolution of attention models for offline and streaming speech recognition within recurrent neural network- and Transformer- based architectures.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 56 of its 91 references that are in the map. First posted on arXiv 14 Feb 2021.

Builds on

56 of 91 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2016Layer Normalizationcited by 5,669
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  10. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  11. 2016Deep Networks with Stochastic Depthcited by 979
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
38 more
  1. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  2. 2015Listen, Attend and Spellcited by 543
  3. 2015Attention-Based Models for Speech Recognitioncited by 401
  4. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  5. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  6. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  7. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 151
  8. 2019An Attentive Survey of Attention Modelscited by 124
  9. 2019Attention in Natural Language Processingcited by 101
  10. 2018Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducercited by 96
  11. 2019Language Modeling with Deep Transformerscited by 96
  12. 2019Monotonic Multihead Attentioncited by 93
  13. 2019Transformer-based Acoustic Modeling for Hybrid Speech Recognitioncited by 88
  14. 2017Online and Linear-Time Attention by Enforcing Monotonic Alignmentscited by 85
  15. 2017Monotonic Chunkwise Attentioncited by 83
  16. 2020Streaming automatic speech recognition with the transformer modelcited by 73
  17. 2019Transformer-Transducer: End-to-End Speech Recognition with Self-Attentioncited by 69
  18. 2014End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Resultscited by 68
  19. 2020Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selectioncited by 49
  20. 2019Very Deep Self-Attention Networks for End-to-End Speech Recognitioncited by 46
  21. 2019Transformers with convolutional context for ASRcited by 44
  22. 2017Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LMcited by 43
  23. 2018Improved training of end-to-end attention models for speech recognitioncited by 36
  24. 2020Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memorycited by 35
  25. 2016Very Deep Convolutional Networks for End-to-End Speech Recognitioncited by 34
  26. 2019Self-Attention Aligner: A Latency-Control End-to-End Model for ASR Using Self-Attention Network and Chunk-Hoppingcited by 33
  27. 2018Self-Attentional Acoustic Modelscited by 29
  28. 2020Transformer-based Online CTC/attention End-to-End Speech Recognition Architecturecited by 28
  29. 2017Towards End-to-End Speech Recognition with Deep Convolutional Neural Networkscited by 24
  30. 2019Self-Attention Transducers for End-to-End Speech Recognitioncited by 24
  31. 2019Synchronous Transformers for End-to-End Speech Recognitioncited by 24
  32. 2018Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinesecited by 20
  33. 2020Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASRcited by 19
  34. 2020Attention based on-device streaming speech recognition with large speech corpuscited by 18
  35. 2020Enhancing Monotonic Multihead Attention for Streaming ASRcited by 16
  36. 2018An Online Attention-based Model for Speech Recognitioncited by 13
  37. 2018A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinesecited by 10
  38. 2020Online Automatic Speech Recognition with Listen, Attend and Spell Modelcited by 10

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 0; 2025: 2.

5 papers.

  1. 2025Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languagescited by 5
  2. 2023On Separate Normalization in Self-supervised Transformerscited by 4
  3. 2022A Simple Structure For Building A Robust Modelcited by 1
  4. 2022Class-Specific Channel Attention for Few-Shot Learningcited by 0
  5. 2025Causal Structure Discovery for Error Diagnostics of Children's ASRcited by 0