mlsanityData to 7 Oct 2026

Conformer: Convolution-augmented Transformer for Speech Recognition

arXiv 2005.08100 ↗First posted 16 May 2020eess.AS cs.LG cs.SDcited by 1,838

Not on the map. Why?

Cited by 41 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently Transformer and Convolution neural network (CNN) based models have shown promising results in Automatic Speech Recognition (ASR), outperforming Recurrent neural networks (RNNs). Transformer models are good at capturing content-based global interactions, while CNNs exploit local features effectively. In this work, we achieve the best of both worlds by studying how to combine convolution neural networks and transformers to model both local and global dependencies of an audio sequence in a parameter-efficient way. To this regard, we propose the convolution-augmented transformer for speech recognition, named Conformer. Conformer significantly outperforms the previous Transformer and CNN based models achieving state-of-the-art accuracies. On the widely used LibriSpeech benchmark, our model achieves WER of 2.1%/4.3% without using a language model and 1.9%/3.9% with an external language model on test/testother. We also observe competitive performance of 2.7%/6.3% with a small model of only 10M parameters.

Key facts. Cited by 1,838 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 209; 2022: 350; 2023: 419; 2024: 368; 2025: 305; 2026: 187.

1,838 papers.

Sort:

By area:

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  5. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  6. 2022Transformers learn in-context by gradient descentcited by 714
  7. 2021AST: Audio Spectrogram Transformercited by 634
  8. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  9. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  10. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  11. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  12. 2021A Survey of Transformerscited by 399
38 more
  1. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  2. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  3. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  4. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  5. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  6. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  7. 2021Mesh Graphormercited by 264
  8. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  9. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  10. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  11. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  12. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  13. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  14. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  15. 2026Gemma 4 Technical Reportcited by 187
  16. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  17. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 158
  18. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  19. 2024The Road Less Scheduledcited by 158
  20. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  21. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  22. 2021Unsupervised Speech Recognitioncited by 147
  23. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  24. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 139
  25. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  26. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  27. 2022MetaFormer Baselines for Visioncited by 131
  28. 2023End-to-End Speech Recognition: A Surveycited by 127
  29. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  30. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 123
  31. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  32. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  33. 2022Is Attention All That NeRF Needs?cited by 113
  34. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  35. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  36. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  37. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  38. 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 102

50 most cited of 1,838 All 1,838 (JSON)