mlsanityData to 7 Oct 2026

A Comparative Study on Transformer vs RNN in Speech Applications

arXiv 1909.06317 ↗First posted 13 Sep 2019cs.CL cs.SD eess.AScited by 210

Not on the map. Why?

Abstract (arXiv)

Sequence-to-sequence models have been widely used in end-to-end speech processing, for example, automatic speech recognition (ASR), speech translation (ST), and text-to-speech (TTS). This paper focuses on an emergent sequence-to-sequence model called Transformer, which achieves state-of-the-art performance in neural machine translation and other natural language processing applications. We undertook intensive studies in which we experimentally compared and analyzed Transformer and conventional recurrent neural networks (RNN) in a total of 15 ASR, one multilingual ASR, one ST, and two TTS benchmarks. Our experiments revealed various training tips and significant performance benefits obtained with Transformer for each task including the surprising superiority of Transformer in 13/15 ASR benchmarks in comparison with RNN. We are preparing to release Kaldi-style reproducible recipes using open source and publicly available datasets for all the ASR, ST, and TTS tasks for the community to succeed our exciting outcomes.

Key facts. Cited by 210 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 76; 2022: 50; 2023: 44; 2024: 20; 2025: 15; 2026: 5.

210 papers.

Sort:

By area:

  1. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  2. 2021Going deeper with Image Transformerscited by 759
  3. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  4. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  5. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  6. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  7. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  8. 2023End-to-End Speech Recognition: A Surveycited by 127
  9. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 95
  10. 2022Three things everyone should know about Vision Transformerscited by 90
  11. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  12. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 90
38 more
  1. 2021OadTR: Online Action Detection with Transformerscited by 83
  2. 2023Zipformer: A faster and better encoder for automatic speech recognitioncited by 77
  3. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  4. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  5. 2023A Text-guided Protein Design Frameworkcited by 61
  6. 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
  7. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 50
  8. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  9. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  10. 2023Transformers in Speech Processing: A Surveycited by 38
  11. 2021Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognitioncited by 37
  12. 2021Momentum Pseudo-Labeling for Semi-Supervised Speech Recognitioncited by 37
  13. 2021End-to-End Speaker-Attributed ASR with Transformercited by 35
  14. 2021Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformercited by 34
  15. 2021Exploiting Adapters for Cross-lingual Low-resource Speech Recognitioncited by 33
  16. 2021Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERTcited by 31
  17. 2022Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Datacited by 30
  18. 2021Transformer Based Deliberation for Two-Pass Speech Recognitioncited by 29
  19. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
  20. 2021A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generationcited by 27
  21. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  22. 2022ASR2K: Speech Recognition for Around 2000 Languages without Audiocited by 26
  23. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 26
  24. 2021Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformerscited by 25
  25. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  26. 2023Audio-Visual Efficient Conformer for Robust Speech Recognitioncited by 22
  27. 2021Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognitioncited by 19
  28. 2022AISHELL-NER: Named Entity Recognition from Chinese Speechcited by 19
  29. 2024Exploring the Capability of Mamba in Speech Applicationscited by 19
  30. 2025Foundation Models for Time Series: A Surveycited by 19
  31. 2021An evaluation of word-level confidence estimation for end-to-end automatic speech recognitioncited by 18
  32. 2021A Survey on Deep Reinforcement Learning for Audio-Based Applicationscited by 17
  33. 2021Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognitioncited by 17
  34. 2022Accelerating RNN-T Training and Inference Using CTC guidancecited by 17
  35. 2021SpeechNet: A Universal Modularized Model for Speech Processing Taskscited by 16
  36. 2021Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognitioncited by 16
  37. 2021Dyadformer: A Multi-modal Transformer for Long-Range Modeling of Dyadic Interactionscited by 16
  38. 2023SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and Effective Hotword Customization Abilitycited by 16

50 most cited of 210 All 210 (JSON)