mlsanityData to 7 Oct 2026

A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks

arXiv 2306.07303 ↗First posted 11 Jun 2023cs.LG cs.CLcited by 70

Not on the map. Why?

Abstract (arXiv)

Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. Unlike conventional neural networks or updated versions of Recurrent Neural Networks (RNNs) such as Long Short-Term Memory (LSTM), transformer models excel in handling long dependencies between input sequence elements and enable parallel processing. As a result, transformer-based models have attracted substantial interest among researchers in the field of artificial intelligence. This can be attributed to their immense potential and remarkable achievements, not only in Natural Language Processing (NLP) tasks but also in a wide range of domains, including computer vision, audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published highlighting the transformer's contributions in specific fields, architectural differences, or performance evaluations, there is still a significant absence of a comprehensive survey paper encompassing its major applications across various domains. Therefore, we undertook the task of filling this gap by conducting an extensive survey of proposed transformer models from 2017 to 2022. Our survey encompasses the identification of the top five application domains for transformer-based models, namely: NLP, Computer Vision, Multi-Modality, Audio and Speech Processing, and Signal Processing. We analyze the impact of highly influential transformer-based models in these domains and subsequently classify them based on their respective tasks using a proposed taxonomy. Our aim is to shed light on the existing potential and future possibilities of transformers for enthusiastic researchers, thus contributing to the broader understanding of this groundbreaking technology.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 134 of its 240 references that are in the map. First posted on arXiv 11 Jun 2023.

Builds on

134 of 240 references.

References · 134

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  13. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  14. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  15. 2021ViViT: A Video Vision Transformercited by 1,610
  16. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  17. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  18. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  19. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  20. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  21. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  22. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  23. 2021Transformers in Vision: A Surveycited by 1,009
  24. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  25. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  26. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  27. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  28. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  29. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  30. 2020Efficient Transformers: A Surveycited by 784
  31. 2018Image Transformercited by 770
  32. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  33. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
  34. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  35. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  36. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  37. 2020A Survey on Visual Transformercited by 653
  38. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651

50 most cited of 134 All 134 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 24; 2025: 36; 2026: 9.

70 papers.

Sort:

By area:

  1. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  2. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  3. 2024Your Transformer is Secretly Linearcited by 23
  4. 2024Classification, Regression and Segmentation directly from k-Space in Cardiac MRIcited by 8
  5. 2024Unveiling Interpretability in Self-Supervised Speech Representations for Parkinson's Diagnosiscited by 8
  6. 2024Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategycited by 7
  7. 2025STEER-ME: Assessing the Microeconomic Reasoning of Large Language Modelscited by 7
  8. 2024Attention as an RNNcited by 6
  9. 2025Algebraic Language Models for Inverse Design of Metamaterials via Diffusion Transformerscited by 5
  10. 2024Enhancing IoT Intelligence: A Transformer-based Reinforcement Learning Methodologycited by 4
  11. 2025T-SHRED: Symbolic Regression for Regularization and Model Discovery with Transformer Shallow Recurrent Decoderscited by 4
  12. 2025MCP-Orchestrated Multi-Agent System for Automated Disinformation Detectioncited by 4
38 more
  1. 2024The last Dance : Robust backdoor attack via diffusion models and bayesian approachcited by 3
  2. 2024FViT: A Focal Vision Transformer with Gabor Filtercited by 3
  3. 2024Trading Devil: Robust backdoor attack via Stochastic investment models and Bayesian approachcited by 3
  4. 2024Survey and Taxonomy: The Role of Data-Centric AI in Transformer-Based Time Series Forecastingcited by 3
  5. 2024Performance Evaluation of Tokenizers in Large Language Models for the Assamese Languagecited by 3
  6. 2025Natural Language Generation in Healthcare: A Review of Methods and Applicationscited by 3
  7. 2025Transformer-based EEG Decoding: A Surveycited by 3
  8. 2025Differentiable Hierarchical Visual Tokenizationcited by 3
  9. 2024PagPassGPT: Pattern Guided Password Guessing via Generative Pretrained Transformercited by 2
  10. 2024Advances in Transformers for Robotic Applications: A Reviewcited by 2
  11. 2025Automatic selection of the best neural architecture for time series forecastingcited by 2
  12. 2025NeuroAMP: A Novel End-to-end General Purpose Deep Neural Amplifier for Personalized Hearing Aidscited by 2
  13. 2025Integrating Biological and Machine Intelligence: Attention Mechanisms in Brain-Computer Interfacescited by 2
  14. 2025Simulation-based inference for Precision Neutrino Physics through Neural Monte Carlo tuningcited by 2
  15. 2025Hybrid LSTM-Transformer Models for Profiling Highway-Railway Grade Crossingscited by 2
  16. 2026Affine-Scaled Attention: Towards Flexible and Stable Transformer Attentioncited by 2
  17. 2024Adaptive Point Transformercited by 1
  18. 2024Short-Term Photovoltaic Forecasting Model for Qualifying Uncertainty during Hazy Weathercited by 1
  19. 2024Optimizing Vision Transformers with Data-Free Knowledge Transfercited by 1
  20. 2024One-Stage-TFS: Thai One-Stage Fingerspelling Dataset for Fingerspelling Recognition Frameworkscited by 1
  21. 2025A Hybrid Swarm Intelligence Approach for Optimizing Multimodal Large Language Models Deployment in Edge-Cloud-based Federated Learning Environmentscited by 1
  22. 2025ChordFormer: A Conformer-Based Architecture for Large-Vocabulary Audio Chord Recognitioncited by 1
  23. 2025Transformer-Empowered Actor-Critic Reinforcement Learning for Sequence-Aware Service Function Chain Partitioningcited by 1
  24. 2025Comparative Evaluation of ChatGPT and DeepSeek Across Key NLP Tasks: Strengths, Weaknesses, and Domain-Specific Performancecited by 1
  25. 2025Calibrated Self-supervised Vision Transformers Improve Intracranial Arterial Calcification Segmentation from Clinical CT Head Scanscited by 1
  26. 2025Probing Information Distribution in Transformer Architectures through Entropy Analysiscited by 1
  27. 2025A Fisher's exact test justification of the TF-IDF term-weighting schemecited by 1
  28. 2025Dynamic Reasoning Chains through Depth-Specialized Mixture-of-Experts in Transformer Architecturescited by 1
  29. 2025The Outputs of Large Language Models are Meaninglesscited by 1
  30. 2026MixQuant: Adaptive Mixed-Precision Quantization for Large Language Modelscited by 1
  31. 2023A Survey on Design Methodologies for Accelerating Deep Learning on Heterogeneous Architecturescited by 0
  32. 2024Trading Devil Final: Backdoor attack via Stock market and Bayesian Optimizationcited by 0
  33. 2024Macformer: Transformer with Random Maclaurin Feature Attentioncited by 0
  34. 2024Frame forecasting in cine MRI using the PCA respiratory motion model: comparing recurrent neural networks trained online and transformerscited by 0
  35. 2024Machine Learning Innovations in CPR: A Comprehensive Survey on Enhanced Resuscitation Techniquescited by 0
  36. 2024Trading Devil RL: Backdoor attack via Stock market, Bayesian Optimization and Reinforcement Learningcited by 0
  37. 2025Deep Causal Behavioral Policy Learning: Applications to Healthcarecited by 0
  38. 2025On the Limits of Applying Graph Transformers for Brain Connectome Classificationcited by 0

50 most cited of 70 All 70 (JSON)