mlsanityData to 7 Oct 2026

Transformers in Speech Processing: A Survey

arXiv 2303.11607 ↗First posted 21 Mar 2023cs.CL cs.SD eess.AScited by 38

Not on the map. Why?

Abstract (arXiv)

The remarkable success of transformers in the field of natural language processing has sparked the interest of the speech-processing community, leading to an exploration of their potential for modeling long-range dependencies within speech sequences. Recently, transformers have gained prominence across various speech-related domains, including automatic speech recognition, speech synthesis, speech translation, speech para-linguistics, speech enhancement, spoken dialogue systems, and numerous multimodal applications. In this paper, we present a comprehensive survey that aims to bridge research studies from diverse subfields within speech technology. By consolidating findings from across the speech technology landscape, we provide a valuable resource for researchers interested in harnessing the power of transformers to advance the field. We identify the challenges encountered by transformers in speech processing while also offering insights into potential solutions to address these issues.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 237 of its 319 references that are in the map. First posted on arXiv 21 Mar 2023.

Builds on

237 of 319 references.

References · 237

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  13. 2020Rethinking Attention with Performerscited by 1,584
  14. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  15. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  16. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  17. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  18. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  19. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  20. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  21. 2021Transformers in Vision: A Surveycited by 1,009
  22. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  23. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  24. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  25. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  26. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  27. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  28. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  29. 2020Efficient Transformers: A Surveycited by 784
  30. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  31. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  32. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  33. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  34. 2014Distributed Representations of Sentences and Documentscited by 677
  35. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  36. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  37. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  38. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596

50 most cited of 237 All 237 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 14; 2025: 6; 2026: 3.

38 papers.

Sort:

By area:

  1. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  2. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  3. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 30
  4. 2023Transformers in Healthcare: A Surveycited by 17
  5. 2024HybridFlow: Infusing Continuity into Masked Codebook for Extreme Low-Bitrate Image Compressioncited by 12
  6. 2023Graph Convolutions Enrich the Self-Attention in Transformers!cited by 11
  7. 2023Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformerscited by 9
  8. 2024LifeGPT: Topology-Agnostic Generative Pretrained Transformer Model for Cellular Automatacited by 8
  9. 2024Automatic Speech Recognition with BERT and CTC Transformers: A Reviewcited by 8
  10. 2024Scaling Sequential Recommendation Models with Transformerscited by 7
  11. 2023Knowledge Relation Rank Enhanced Heterogeneous Learning Interaction Modeling for Neural Graph Forgetting Knowledge Tracingcited by 6
  12. 2023A Preliminary Study on Augmenting Speech Emotion Recognition using a Diffusion Modelcited by 6
26 more
  1. 2024Attention as an RNNcited by 6
  2. 2024A Survey of Deep Learning Audio Generation Methodscited by 5
  3. 2023ivrit.ai: A Comprehensive Dataset of Hebrew Speech for AI Research and Developmentcited by 4
  4. 2025Provable In-Context Learning of Nonlinear Regression with Transformerscited by 4
  5. 2023Active Bird2Vec: Towards End-to-End Bird Sound Monitoring with Transformerscited by 3
  6. 2023A Lightweight Transformer for Faster and Robust EBSD Data Collectioncited by 3
  7. 2024Federated and Transfer Learning for Cancer Detection Based on Image Analysiscited by 3
  8. 2023Vulnerability Analysis of Transformer-based Optical Character Recognition to Adversarial Attackscited by 2
  9. 2025AutomataGPT: Forecasting and Ruleset Inference for Two-Dimensional Cellular Automatacited by 2
  10. 2023Cross-Language Speech Emotion Recognition Using Multimodal Dual Attention Transformerscited by 1
  11. 2023Attention Is Not All You Need Anymorecited by 1
  12. 2024How do Hyenas deal with Human Speech? Speech Recognition and Translation with ConfHyenacited by 1
  13. 2024Cognitively Inspired Energy-Based World Modelscited by 1
  14. 2025IML-Spikeformer: Input-aware Multi-Level Spiking Transformer for Speech Processingcited by 1
  15. 2023Inspecting Spoken Language Understanding from Kids for Basic Math Learning at Homecited by 0
  16. 2023Echotune: A Modular Extractor Leveraging the Variable-Length Nature of Speech in ASR Taskscited by 0
  17. 2024Child Speech Recognition in Human-Robot Interaction: Problem Solved?cited by 0
  18. 2024DemOpts: Fairness corrections in COVID-19 case prediction modelscited by 0
  19. 2024Optimizing Multi-Stuttered Speech Classification: Leveraging Whisper's Encoder for Efficient Parameter Reduction in Automated Assessmentcited by 0
  20. 2024Survey: Transformer-based Models in Data Modality Conversioncited by 0
  21. 2024AttentionX: Exploiting Consensus Discrepancy In Attention from A Distributed Optimization Perspectivecited by 0
  22. 2025Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspectivecited by 0
  23. 2025Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problemscited by 0
  24. 2026ADVOSYNTH: A Synthetic Multi-Advocate Dataset for Speaker Identification in Courtroom Scenarioscited by 0
  25. 2026LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Modelscited by 0
  26. 2026Improved Belief-Attention in Vision Taskcited by 0